Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Matthew Knepley]

On Fri, Oct 27, 2023 at 3:54 PM Damyn Chipman 
wrote:

> Yeah, I’ll make an issue and use a modified version of this test routine.
>
> Does anything change if I will be using MATSCALAPACK matrices instead of
> the built in MATDENSE?
>

No, that is likely worse.


> Like I said, I will be computing Schur complements and need to use a
> parallel and dense matrix format.
>

I do not understand the communication pattern, but it is possible that
Elemental would be slightly faster since it has some cool built-in
communication operations, however it might be more programming.

  Thanks,

 Matt


> -Damyn
>
> On Oct 26, 2023, at 10:01 AM, Matthew Knepley  wrote:
>
> On Wed, Oct 25, 2023 at 11:55 PM Damyn Chipman <
> damynchip...@u.boisestate.edu> wrote:
>
>> Great thanks, that seemed to work well. This is something my algorithm
>> will do fairly often (“elevating” a node’s communicator to a communicator
>> that includes siblings). The matrices formed are dense but low rank. With
>> MatCreateSubMatrix, it appears I do a lot of copying from one Mat to
>> another. Is there a way to do it with array copying or pointer movement
>> instead of copying entries?
>>
>
> We could make a fast path for dense that avoids MatSetValues(). Can you
> make an issue for this? The number one thing that would make this faster is
> to contribute a small test. Then we could run it continually when putting
> in the fast path to make sure we are preserving correctness.
>
>   Thanks,
>
> Matt
>
>
>> -Damyn
>>
>> On Oct 24, 2023, at 9:51 PM, Jed Brown  wrote:
>>
>> You can place it in a parallel Mat (that has rows or columns on only one
>> rank or a subset of ranks) and then MatCreateSubMatrix with all new
>> rows/columns on a different rank or subset of ranks.
>>
>> That said, you usually have a function that assembles the matrix and you
>> can just call that on the new communicator.
>>
>> Damyn Chipman  writes:
>>
>> Hi PETSc developers,
>>
>> In short, my question is this: Does PETSc provide a way to move or copy
>> an object (say a Mat) from one communicator to another?
>>
>> The more detailed scenario is this: I’m working on a linear algebra
>> solver on quadtree meshes (i.e., p4est). I use communicator subsets in
>> order to facilitate communication between siblings or nearby neighbors.
>> When performing linear algebra across siblings (a group of 4), I need to
>> copy a node’s data (i.e., a Mat object) from a sibling’s communicator to
>> the communicator that includes the four siblings. From what I can tell, I
>> can only copy a PETSc object onto the same communicator.
>>
>> My current approach will be to copy the raw data from the Mat on one
>> communicator to a new Mat on the new communicator, but I wanted to see if
>> there is a more “elegant” approach within PETSc.
>>
>> Thanks in advance,
>>
>> Damyn Chipman
>> Boise State University
>> PhD Candidate
>> Computational Sciences and Engineering
>> damynchip...@u.boisestate.edu
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] [petsc-maint] DMSwarm on multiple processors

[Matthew Knepley]

Okay, there were a few problems:

1) You overwrote the bounds on string loc_grid_gen[]

2) You destroyed the coordinate DA

I fixed these and it runs for me fine on several processes. I am including
my revised source since I check a lot more error values. I converted it to
C because that is easier for me, although C has a problem with your sqrt()
in a compile-time constant.

  Thanks,

 Matt

On Thu, Oct 26, 2023 at 10:59 AM Barry Smith  wrote:

>
>Please run with -malloc_debug option or even better run under Valgrind
> https://petsc.org/release/faq/
>
>
>
> On Oct 26, 2023, at 10:35 AM, Joauma Marichal via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Hello,
>
> Here is a very simple version where I have issues.
>
> Which I run as follows:
>
> cd Grid_generation
> make clean
> make all
> ./grid_generation
> cd ..
> make clean
> make all
> ./cobpor # on 1 proc
> # OR
> mpiexec ./cobpor -ksp_type cg -pc_type pfmg -dm_mat_type hyprestruct
> -pc_pfmg_skip_relax 1 -pc_pfmg_rap_time non-Galerkin # on multiple procs
>
> The error that I get is the following:
> munmap_chunk(): invalid pointer
> [cns266:2552391] *** Process received signal ***
> [cns266:2552391] Signal: Aborted (6)
> [cns266:2552391] Signal code:  (-6)
> [cns266:2552391] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7fd7fd194b20]
> [cns266:2552391] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7fd7fd194a9f]
> [cns266:2552391] [ 2] /lib64/libc.so.6(abort+0x127)[0x7fd7fd167e05]
> [cns266:2552391] [ 3] /lib64/libc.so.6(+0x91037)[0x7fd7fd1d7037]
> [cns266:2552391] [ 4] /lib64/libc.so.6(+0x9819c)[0x7fd7fd1de19c]
> [cns266:2552391] [ 5] /lib64/libc.so.6(+0x9844c)[0x7fd7fd1de44c]
> [cns266:2552391] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(PetscFreeAlign+0xe)[0x7fd7fe63d50e]
> [cns266:2552391] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(DMSetMatType+0x3d)[0x7fd7feab87ad]
> [cns266:2552391] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(DMSetFromOptions+0x109)[0x7fd7feab8b59]
> [cns266:2552391] [ 9] ./cobpor[0x402df9]
> [cns266:2552391] [10] /lib64/libc.so
> .6(__libc_start_main+0xf3)[0x7fd7fd180cf3]
> [cns266:2552391] [11] ./cobpor[0x40304e]
> [cns266:2552391] *** End of error message ***
>
>
> Thanks a lot for your help.
>
> Best regards,
>
> Joauma
>
>
>
>
> *De : *Matthew Knepley 
> *Date : *mercredi, 25 octobre 2023 à 14:45
> *À : *Joauma Marichal 
> *Cc : *petsc-ma...@mcs.anl.gov ,
> petsc-users@mcs.anl.gov 
> *Objet : *Re: [petsc-maint] DMSwarm on multiple processors
> On Wed, Oct 25, 2023 at 8:32 AM Joauma Marichal via petsc-maint <
> petsc-ma...@mcs.anl.gov> wrote:
>
> Hello,
>
> I am using the DMSwarm library in some Eulerian-Lagrangian approach to
> have vapor bubbles in water.
> I have obtained nice results recently and wanted to perform bigger
> simulations. Unfortunately, when I increase the number of processors used
> to run the simulation, I get the following error:
>
>
> free(): invalid size
>
> [cns136:590327] *** Process received signal ***
>
> [cns136:590327] Signal: Aborted (6)
>
> [cns136:590327] Signal code:  (-6)
>
> [cns136:590327] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f56cd4c9b20]
>
> [cns136:590327] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f56cd4c9a9f]
>
> [cns136:590327] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f56cd49ce05]
>
> [cns136:590327] [ 3] /lib64/libc.so.6(+0x91037)[0x7f56cd50c037]
>
> [cns136:590327] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f56cd51319c]
>
> [cns136:590327] [ 5] /lib64/libc.so.6(+0x99aac)[0x7f56cd514aac]
>
> [cns136:590327] [ 6] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71e64]
>
> [cns136:590327] [ 7] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(+0x841642)[0x7f56cea83642]
>
> [cns136:590327] [ 8] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(PetscSFSetUp+0x9e)[0x7f56cea7043e]
>
> [cns136:590327] [ 9] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(VecScatterCreate+0x164e)[0x7f56cea7bbde]
>
> [cns136:590327] [10] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(DMSetUp_DA_3D+0x3e38)[0x7f56cee84dd8]
>
> [cns136:590327] [11] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(DMSetUp_DA+0xd8)[0x7f56cee9b448]
>
> [cns136:590327] [12] /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/
> libpetsc.so.3.019(DMSetUp+0x20)[0x7f56cededa20]
>
> [cns136:590327] [13] ./cobpor[0x4418dc]
>
> [cns136:590327] [14] ./cobpor[0x408b63]
>
> 

Re: [petsc-users] Copying PETSc Objects Across MPI Communicators

[Matthew Knepley]

On Wed, Oct 25, 2023 at 11:55 PM Damyn Chipman <
damynchip...@u.boisestate.edu> wrote:

> Great thanks, that seemed to work well. This is something my algorithm
> will do fairly often (“elevating” a node’s communicator to a communicator
> that includes siblings). The matrices formed are dense but low rank. With
> MatCreateSubMatrix, it appears I do a lot of copying from one Mat to
> another. Is there a way to do it with array copying or pointer movement
> instead of copying entries?
>

We could make a fast path for dense that avoids MatSetValues(). Can you
make an issue for this? The number one thing that would make this faster is
to contribute a small test. Then we could run it continually when putting
in the fast path to make sure we are preserving correctness.

  Thanks,

Matt


> -Damyn
>
> On Oct 24, 2023, at 9:51 PM, Jed Brown  wrote:
>
> You can place it in a parallel Mat (that has rows or columns on only one
> rank or a subset of ranks) and then MatCreateSubMatrix with all new
> rows/columns on a different rank or subset of ranks.
>
> That said, you usually have a function that assembles the matrix and you
> can just call that on the new communicator.
>
> Damyn Chipman  writes:
>
> Hi PETSc developers,
>
> In short, my question is this: Does PETSc provide a way to move or copy an
> object (say a Mat) from one communicator to another?
>
> The more detailed scenario is this: I’m working on a linear algebra solver
> on quadtree meshes (i.e., p4est). I use communicator subsets in order to
> facilitate communication between siblings or nearby neighbors. When
> performing linear algebra across siblings (a group of 4), I need to copy a
> node’s data (i.e., a Mat object) from a sibling’s communicator to the
> communicator that includes the four siblings. From what I can tell, I can
> only copy a PETSc object onto the same communicator.
>
> My current approach will be to copy the raw data from the Mat on one
> communicator to a new Mat on the new communicator, but I wanted to see if
> there is a more “elegant” approach within PETSc.
>
> Thanks in advance,
>
> Damyn Chipman
> Boise State University
> PhD Candidate
> Computational Sciences and Engineering
> damynchip...@u.boisestate.edu
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] OpenMP doesn't work anymore with PETSc building rules

[Matthew Knepley]

On Wed, Oct 25, 2023 at 11:12 AM Qiyue Lu  wrote:

> Hello,
> I have an in-house code enabled OpenMP and it works. Now I am trying to
> incorporate PETSc as the linear solver and build together using the
> building rules in $PETSC_HOME/lib/petsc/conf/rules. However, I found the
> OpenMP part doesn't work anymore.
> Should I re-configure the petsc installation with --with-openmp=1 option?
> I wonder are the building rules affected by this missing option?
>

There are parts of PETSc that are not threadsafe unless you configure using
--with-threadsafety. If you plan to call PETSc methods from different
threads, you need this.

  Thanks,

 Matt


> Thanks,
> Qiyue Lu
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] [petsc-maint] DMSwarm on multiple processors

[Matthew Knepley]

On Wed, Oct 25, 2023 at 8:32 AM Joauma Marichal via petsc-maint <
petsc-ma...@mcs.anl.gov> wrote:

> Hello,
>
>
>
> I am using the DMSwarm library in some Eulerian-Lagrangian approach to
> have vapor bubbles in water.
>
> I have obtained nice results recently and wanted to perform bigger
> simulations. Unfortunately, when I increase the number of processors used
> to run the simulation, I get the following error:
>
>
>
> free(): invalid size
>
> [cns136:590327] *** Process received signal ***
>
> [cns136:590327] Signal: Aborted (6)
>
> [cns136:590327] Signal code:  (-6)
>
> [cns136:590327] [ 0] /lib64/libc.so.6(+0x4eb20)[0x7f56cd4c9b20]
>
> [cns136:590327] [ 1] /lib64/libc.so.6(gsignal+0x10f)[0x7f56cd4c9a9f]
>
> [cns136:590327] [ 2] /lib64/libc.so.6(abort+0x127)[0x7f56cd49ce05]
>
> [cns136:590327] [ 3] /lib64/libc.so.6(+0x91037)[0x7f56cd50c037]
>
> [cns136:590327] [ 4] /lib64/libc.so.6(+0x9819c)[0x7f56cd51319c]
>
> [cns136:590327] [ 5] /lib64/libc.so.6(+0x99aac)[0x7f56cd514aac]
>
> [cns136:590327] [ 6]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUpRanks+0x4c4)[0x7f56cea71e64]
>
> [cns136:590327] [ 7]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(+0x841642)[0x7f56cea83642]
>
> [cns136:590327] [ 8]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(PetscSFSetUp+0x9e)[0x7f56cea7043e]
>
> [cns136:590327] [ 9]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(VecScatterCreate+0x164e)[0x7f56cea7bbde]
>
> [cns136:590327] [10]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA_3D+0x3e38)[0x7f56cee84dd8]
>
> [cns136:590327] [11]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp_DA+0xd8)[0x7f56cee9b448]
>
> [cns136:590327] [12]
> /gpfs/home/acad/ucl-tfl/marichaj/marha/lib_petsc/lib/libpetsc.so.3.019(DMSetUp+0x20)[0x7f56cededa20]
>
> [cns136:590327] [13] ./cobpor[0x4418dc]
>
> [cns136:590327] [14] ./cobpor[0x408b63]
>
> [cns136:590327] [15]
> /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f56cd4b5cf3]
>
> [cns136:590327] [16] ./cobpor[0x40bdee]
>
> [cns136:590327] *** End of error message ***
>
> --
>
> Primary job  terminated normally, but 1 process returned
>
> a non-zero exit code. Per user-direction, the job has been aborted.
>
> --
>
> --
>
> mpiexec noticed that process rank 84 with PID 590327 on node cns136 exited
> on signal 6 (Aborted).
>
> --
>
>
>
> When I reduce the number of processors the error disappears and when I run
> my code without the vapor bubbles it also works.
>
> The problem seems to take place at this moment:
>
>
>
> DMCreate(PETSC_COMM_WORLD,swarm);
>
> DMSetType(*swarm,DMSWARM);
>
> DMSetDimension(*swarm,3);
>
> DMSwarmSetType(*swarm,DMSWARM_PIC);
>
> DMSwarmSetCellDM(*swarm,*dmcell);
>
>
>
>
>
> Thanks a lot for your help.
>

Things that would help us track this down:

1) The smallest example where it fails

2) The smallest number of processes where it fails

3) A stack trace of the failure

4) A simple example that we can run that also fails

  Thanks,

 Matt


> Best regards,
>
>
>
> Joauma
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Seeking Clarification on SNES Solver Behavior

[Matthew Knepley]

On Mon, Oct 23, 2023 at 10:23 PM Rene Chenard via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi!
>
> We have recently noticed some inconsistencies in the behavior of the SNES
> solver when using different solver types, and we would greatly appreciate
> your insights in resolving this matter.
>
> While working with SNESSolve in parallel, we have encountered a
> discrepancy in the behavior of the evaluation functions for the
> ComputeFunction and the JacobianFunction. Specifically, there seems to be
> an inconsistency in whether Vec x receives automatic updates to its ghosts
> or if manual updates are required (with calls to VecGhostUpdateBegin/End).
>
> For instance, when using the ngmres solver, the ghosts of Vec x are
> adequately updated. However, when employing the nrichardson solver, it
> appears that manual updates to the ghosts are necessary.
>
> It's important to note that we do not utilize the DM object in our
> implementation, as we have developed our own solution to manage models and
> discretization.
>
> To better understand the root cause of this behavior, we kindly request
> your assistance in determining if we may be overlooking something in our
> implementation, or if there are inherent inconsistencies in the SNES solver
> itself.
>
> Your expertise in this matter would be invaluable to us, and we thank you
> in advance for your consideration and support.
>

Since you are not using DM, does that mean that you register a callback with

  https://petsc.org/main/manualpages/SNES/DMSNESSetFunction/

If so, we do not do any kind of local-to-global calls. I am not sure how
NGMRES would populate local
vectors for you. My guess is that you have a ghost update call somewhere in
your callback, and this
gets hit in the NGMRES because it has extra residual evaluations.

We can be more specific with more details about the code.

   Thanks,

 Matt



> Warm regards,
>
> —René Chenard
> Research Professional at Université Laval
> rene.chenar...@ulaval.ca
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Performance of Conda Binary vs Self Compiled Version

[Matthew Knepley]

On Thu, Oct 19, 2023 at 8:35 PM Jorge Nin  wrote:

> Hi Mathew,
>
> Thanks for the response. It actually seems like the matrix is very sparse 
> (0.99%
> sparsity from what I’m measuring). It’s an FEA solver so it would make
> sense.
> My current guess is the optimization flags are making a large difference
> for the M1 Mac, but I am also surprised it makes such a huge difference.
>
> It’s why I was asking if there was a resource or another to use my own
> version of PETSc with Conda.
>

We do not know how Conda works unfortunately.

  Thanks,

 Matt


> I believe a 2-3 x speed up is worth the hassle.
>
>
> Best,
> Jorge
>
>
>
> On Oct 19, 2023, at 4:00 PM, Matthew Knepley  wrote:
>
> On Thu, Oct 19, 2023 at 3:54 PM Jorge Nin  wrote:
>
>> Hi,
>> I was playing around with a self compiled version and, and a the Conda
>> binary of Petsc on the same problem, on my M1 Mac.
>> Interestingly I found that the Conda binary solves the problem 2-3 times
>> slower vs the self compiled version. (For context I’m using the petsc4py
>> python interface)
>>
>> I’ve attached two log views to show the comparison.
>>
>> I was mostly curious about the possible cause for this.
>>
>
> All the time is in the LU numeric factorization. I don't know if your
> matrix is sparse or dense. I am guessing it is dense and different LAPACK
> implementations are linked. If it is sparse, then the compiler options are
> different between builds, but I would be surprised if it made this much
> difference.
>
>   Thanks,
>
>  Matt
>
>
>>  I was also curious how I could use my own compiled version of PETSc in
>> my Conda install?
>>
>>
>> Best,
>> Jorge
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Performance of Conda Binary vs Self Compiled Version

[Matthew Knepley]

On Thu, Oct 19, 2023 at 3:54 PM Jorge Nin  wrote:

> Hi,
> I was playing around with a self compiled version and, and a the Conda
> binary of Petsc on the same problem, on my M1 Mac.
> Interestingly I found that the Conda binary solves the problem 2-3 times
> slower vs the self compiled version. (For context I’m using the petsc4py
> python interface)
>
> I’ve attached two log views to show the comparison.
>
> I was mostly curious about the possible cause for this.
>

All the time is in the LU numeric factorization. I don't know if your
matrix is sparse or dense. I am guessing it is dense and different LAPACK
implementations are linked. If it is sparse, then the compiler options are
different between builds, but I would be surprised if it made this much
difference.

  Thanks,

 Matt


>  I was also curious how I could use my own compiled version of PETSc in my
> Conda install?
>
>
> Best,
> Jorge
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Coordinate format internal reordering

[Matthew Knepley]

On Thu, Oct 19, 2023 at 1:46 PM Enrico  wrote:

> Sorry but I don't want another partition, Petsc internally is changing
> the partition. I would like to have the same partition that I have in
> the application. Is the example not clear?
>

"Petsc internally is changing the partition" This is not correct. PETSc does
not prescribe partitions. I refer to the documentation for the creation of
Vec:

  https://petsc.org/main/manualpages/Vec/VecCreateMPI/

Here the user sets the local and global sizes. Since data is contiguous,
these
completely define the vector, and are under user control.

  Thanks,

 Matt


> On 19/10/2023 19:43, Matthew Knepley wrote:
> > On Thu, Oct 19, 2023 at 1:00 PM Enrico  > <mailto:degreg...@dkrz.de>> wrote:
> >
> > Here is a very very simple reproducer of my problem. It is a fortran
> > program and it has to run with 2 processes.
> >
> >
> > You seem to be saying that you start with one partition of your data,
> > but you would like
> > another partition. For this, you have to initially communicate. For this
> > I would use VecScatter.
> > However, since most data is generated, I would consider not generating
> > my data in that initial
> > distribution.
> >
> > There are many examples in the repository. In the discretization of a
> > PDE, we first divide the domain,
> > then number up each piece, then assemble the linear algebra objects.
> >
> >Thanks,
> >
> >Matt
> >
> > The output is:
> >
> >process 0 : xx_v(1 ) =
>  0.000
> >process 0 : xx_v(2 ) =
>  1.000
> >process 0 : xx_v(3 ) =
>  2.000
> >process 1 : xx_v(1 ) =
>  3.000
> >process 1 : xx_v(2 ) =
>  4.000
> >process 1 : xx_v(3 ) =
>  5.000
> >
> > and I would like to have:
> >
> >process 0 : xx_v(1 ) =
>  2.000
> >process 0 : xx_v(2 ) =
>  3.000
> >process 0 : xx_v(3 ) =
>  4.000
> >process 1 : xx_v(1 ) =
>  0.000
> >process 1 : xx_v(2 ) =
>  1.000
> >process 1 : xx_v(3 ) =
>  5.000
> >
> > How can I do that?
> >
> > program main
> > #include 
> >   use petscksp
> >   implicit none
> >
> >   PetscErrorCode ierr
> >   PetscInt  :: Psize = 6
> >   integer  :: Lsize
> >   PetscInt  :: work_size
> >   PetscInt  :: work_rank
> >   Vec :: b
> >   integer, allocatable, dimension(:) :: glb_index
> >   double precision, allocatable, dimension(:) :: array
> >   PetscScalar, pointer :: xx_v(:)
> >   integer :: i
> >   PetscCount :: csize
> >
> >   CALL PetscInitialize(ierr)
> >
> >   Lsize = 3
> >   csize = Lsize
> >
> >   allocate(glb_index(0:Lsize-1), array(0:Lsize-1))
> >
> >   CALL MPI_Comm_size(PETSC_COMM_WORLD, work_size, ierr);
> >   CALL MPI_Comm_rank(PETSC_COMM_WORLD, work_rank, ierr);
> >   if (work_rank == 0) then
> > glb_index(0) = 2
> > glb_index(1) = 3
> > glb_index(2) = 4
> > array(0) = 2
> > array(1) = 3
> > array(2) = 4
> >   else if (work_rank == 1) then
> > glb_index(0) = 0
> > glb_index(1) = 1
> > glb_index(2) = 5
> > array(0) = 0
> > array(1) = 1
> > array(2) = 5
> >   end if
> >
> >   ! Create and fill rhs vector
> >   CALL VecCreate(PETSC_COMM_WORLD, b, ierr);
> >   CALL VecSetSizes(b, Lsize, Psize, ierr);
> >   CALL VecSetType(b, VECMPI, ierr);
> >
> >   CALL VecSetPreallocationCOO(b, csize, glb_index, ierr)
> >   CALL VecSetValuesCOO(b, array, INSERT_VALUES, ierr)
> >
> >   CALL VecGetArrayReadF90(b, xx_v, ierr)
> >
> >   do i=1,Lsize
> > write(*,*) 'process ', work_rank, ': xx_v(',i,') = '

Re: [petsc-users] Coordinate format internal reordering

[Matthew Knepley]

On Thu, Oct 19, 2023 at 1:00 PM Enrico  wrote:

> Here is a very very simple reproducer of my problem. It is a fortran
> program and it has to run with 2 processes.
>

You seem to be saying that you start with one partition of your data, but
you would like
another partition. For this, you have to initially communicate. For this I
would use VecScatter.
However, since most data is generated, I would consider not generating my
data in that initial
distribution.

There are many examples in the repository. In the discretization of a PDE,
we first divide the domain,
then number up each piece, then assemble the linear algebra objects.

  Thanks,

  Matt


> The output is:
>
>   process 0 : xx_v(1 ) = 0.000
>   process 0 : xx_v(2 ) = 1.000
>   process 0 : xx_v(3 ) = 2.000
>   process 1 : xx_v(1 ) = 3.000
>   process 1 : xx_v(2 ) = 4.000
>   process 1 : xx_v(3 ) = 5.000
>
> and I would like to have:
>
>   process 0 : xx_v(1 ) = 2.000
>   process 0 : xx_v(2 ) = 3.000
>   process 0 : xx_v(3 ) = 4.000
>   process 1 : xx_v(1 ) = 0.000
>   process 1 : xx_v(2 ) = 1.000
>   process 1 : xx_v(3 ) = 5.000
>
> How can I do that?
>
> program main
> #include 
>  use petscksp
>  implicit none
>
>  PetscErrorCode ierr
>  PetscInt  :: Psize = 6
>  integer  :: Lsize
>  PetscInt  :: work_size
>  PetscInt  :: work_rank
>  Vec :: b
>  integer, allocatable, dimension(:) :: glb_index
>  double precision, allocatable, dimension(:) :: array
>  PetscScalar, pointer :: xx_v(:)
>  integer :: i
>  PetscCount :: csize
>
>  CALL PetscInitialize(ierr)
>
>  Lsize = 3
>  csize = Lsize
>
>  allocate(glb_index(0:Lsize-1), array(0:Lsize-1))
>
>  CALL MPI_Comm_size(PETSC_COMM_WORLD, work_size, ierr);
>  CALL MPI_Comm_rank(PETSC_COMM_WORLD, work_rank, ierr);
>  if (work_rank == 0) then
>glb_index(0) = 2
>glb_index(1) = 3
>glb_index(2) = 4
>array(0) = 2
>array(1) = 3
>array(2) = 4
>  else if (work_rank == 1) then
>glb_index(0) = 0
>glb_index(1) = 1
>glb_index(2) = 5
>array(0) = 0
>array(1) = 1
>array(2) = 5
>  end if
>
>  ! Create and fill rhs vector
>  CALL VecCreate(PETSC_COMM_WORLD, b, ierr);
>  CALL VecSetSizes(b, Lsize, Psize, ierr);
>  CALL VecSetType(b, VECMPI, ierr);
>
>  CALL VecSetPreallocationCOO(b, csize, glb_index, ierr)
>  CALL VecSetValuesCOO(b, array, INSERT_VALUES, ierr)
>
>  CALL VecGetArrayReadF90(b, xx_v, ierr)
>
>  do i=1,Lsize
>write(*,*) 'process ', work_rank, ': xx_v(',i,') = ', xx_v(i)
>  end do
>
>  CALL VecRestoreArrayReadF90(b, xx_v, ierr)
>
>  deallocate(glb_index, array)
>  CALL VecDestroy(b,ierr)
>
>  CALL PetscFinalize(ierr)
>
> end program main
>
>
> On 19/10/2023 17:36, Matthew Knepley wrote:
> > On Thu, Oct 19, 2023 at 11:33 AM Enrico  > <mailto:degreg...@dkrz.de>> wrote:
> >
> > The layout is not poor, just the global indices are not
> contiguous,this
> > has nothing to do with the local memory layout which is extremely
> > optimized for different architectures. I can not change the layout
> > anyway because it's a climate model with a million lines of code.
> >
> > I don't understand why Petsc is doing all this MPI communication
> under
> > the hood.
> >
> >
> > I don't think we are communicating under the hood.
> >
> > I mean, it is changing the layout of the application and doing
> > a lot of communication.
> >
> >
> > We do not create the layout. The user creates the data layout when they
> > create a vector or matrix.
> >
> > Is there no way to force the same layout and
> > provide info about how to do the halo exchange? In this way I can
> have
> > the same memory layout and there is no communication when I fill or
> > fetch the vectors and the matrix.
> >
> >
> > Yes, you tell the vector/matrix your data layout when you create it.
&g

Re: [petsc-users] Coordinate format internal reordering

[Matthew Knepley]

On Thu, Oct 19, 2023 at 11:33 AM Enrico  wrote:

> The layout is not poor, just the global indices are not contiguous,this
> has nothing to do with the local memory layout which is extremely
> optimized for different architectures. I can not change the layout
> anyway because it's a climate model with a million lines of code.
>
> I don't understand why Petsc is doing all this MPI communication under
> the hood.


I don't think we are communicating under the hood.


> I mean, it is changing the layout of the application and doing
> a lot of communication.


We do not create the layout. The user creates the data layout when they
create a vector or matrix.


> Is there no way to force the same layout and
> provide info about how to do the halo exchange? In this way I can have
> the same memory layout and there is no communication when I fill or
> fetch the vectors and the matrix.
>

Yes, you tell the vector/matrix your data layout when you create it.

  Thanks,

  Matt


> Cheers,
> Enrico
>
> On 19/10/2023 17:21, Matthew Knepley wrote:
> > On Thu, Oct 19, 2023 at 10:51 AM Enrico  > <mailto:degreg...@dkrz.de>> wrote:
> >
> > In the application the storage is contiguous but the global indexing
> is
> > not. I would like to use AO as a translation layer but I don't
> > understand it.
> >
> >
> > Why would you choose to index differently from your storage?
> >
> > My case is actually simple even if it is in a large application, I
> have
> >
> > Mat A, Vec b and Vec x
> >
> > After calling KSPSolve, I use VecGetArrayReadF90 to get a pointer to
> > the
> > data and they are in the wrong ordering, so for example the first
> > element of the solution array on process 0 belongs to process 1 in
> the
> > application.
> >
> >
> > Again, this seems to be a poor choice of layout. What we typically do is
> > to partition
> > the data into chunks owned by each process first.
> >
> > Is it at this point that I should use the AO translation layer? This
> > would be quite bad, it means to build Mat A and Vec b there is MPI
> > communication and also to get the data of Vec x back in the
> application.
> >
> >
> > If you want to store data that process i updates on process j, this will
> > need communication.
> >
> > Anyway, I've tried to use AOPetscToApplicationPermuteReal on the
> > solution array but it doesn't work as I would like. Is this function
> > suppose to do MPI communication between processes and fetch the
> values
> > of the application ordering?
> >
> >
> > There is no communication here. That function call just changes one
> > integer into another.
> > If you want to update values on another process, we recommend using
> > VecScatter() or
> > MatSetValues(), both of which take global indices and do communication
> > if necessary.
> >
> >Thanks,
> >
> >  Matt
> >
> > Cheers,
> > Enrico
> >
> > On 19/10/2023 15:25, Matthew Knepley wrote:
> >  > On Thu, Oct 19, 2023 at 8:57 AM Enrico  > <mailto:degreg...@dkrz.de>
> >  > <mailto:degreg...@dkrz.de <mailto:degreg...@dkrz.de>>> wrote:
> >  >
> >  > Maybe I wasn't clear enough. I would like to completely get
> > rid of
> >  > Petsc
> >  > ordering because I don't want extra communication between
> > processes to
> >  > construct the vector and the matrix (since I have to fill
> > them every
> >  > time step because I'm just using the linear solver with a Mat
> > and a Vec
> >  > data structure). I don't understand how I can do that.
> >  >
> >  >
> >  > Any program you write to do linear algebra will have contiguous
> > storage
> >  > because it
> >  > is so much faster. Contiguous indexing makes sense for contiguous
> >  > storage. If you
> >  > want to use non-contiguous indexing for contiguous storage, you
> > would
> >  > need some
> >  > translation layer. The AO is such a translation, but you could do
> > this
> >  > any way you want.
> >  >
> >  >Thanks,
> >  >
> >  >   Matt
> >  >
> >  > My initial idea was to create another global index ordering
> > within 

Re: [petsc-users] Coordinate format internal reordering

[Matthew Knepley]

On Thu, Oct 19, 2023 at 10:51 AM Enrico  wrote:

> In the application the storage is contiguous but the global indexing is
> not. I would like to use AO as a translation layer but I don't
> understand it.
>

Why would you choose to index differently from your storage?


> My case is actually simple even if it is in a large application, I have
>
> Mat A, Vec b and Vec x
>
> After calling KSPSolve, I use VecGetArrayReadF90 to get a pointer to the
> data and they are in the wrong ordering, so for example the first
> element of the solution array on process 0 belongs to process 1 in the
> application.
>

Again, this seems to be a poor choice of layout. What we typically do is to
partition
the data into chunks owned by each process first.


> Is it at this point that I should use the AO translation layer? This
> would be quite bad, it means to build Mat A and Vec b there is MPI
> communication and also to get the data of Vec x back in the application.
>

If you want to store data that process i updates on process j, this will
need communication.


> Anyway, I've tried to use AOPetscToApplicationPermuteReal on the
> solution array but it doesn't work as I would like. Is this function
> suppose to do MPI communication between processes and fetch the values
> of the application ordering?
>

There is no communication here. That function call just changes one integer
into another.
If you want to update values on another process, we recommend using
VecScatter() or
MatSetValues(), both of which take global indices and do communication if
necessary.

  Thanks,

Matt


> Cheers,
> Enrico
>
> On 19/10/2023 15:25, Matthew Knepley wrote:
> > On Thu, Oct 19, 2023 at 8:57 AM Enrico  > <mailto:degreg...@dkrz.de>> wrote:
> >
> > Maybe I wasn't clear enough. I would like to completely get rid of
> > Petsc
> > ordering because I don't want extra communication between processes
> to
> > construct the vector and the matrix (since I have to fill them every
> > time step because I'm just using the linear solver with a Mat and a
> Vec
> > data structure). I don't understand how I can do that.
> >
> >
> > Any program you write to do linear algebra will have contiguous storage
> > because it
> > is so much faster. Contiguous indexing makes sense for contiguous
> > storage. If you
> > want to use non-contiguous indexing for contiguous storage, you would
> > need some
> > translation layer. The AO is such a translation, but you could do this
> > any way you want.
> >
> >Thanks,
> >
> >   Matt
> >
> > My initial idea was to create another global index ordering within my
> > application to use only for the Petsc interface but then I think that
> > the ghost cells are wrong.
> >
> > On 19/10/2023 14:50, Matthew Knepley wrote:
> >  > On Thu, Oct 19, 2023 at 6:51 AM Enrico  > <mailto:degreg...@dkrz.de>
> >  > <mailto:degreg...@dkrz.de <mailto:degreg...@dkrz.de>>> wrote:
> >  >
> >  > Hello,
> >  >
> >  > if I create an application ordering using AOCreateBasic,
> should I
> >  > provide the same array for const PetscInt myapp[] and const
> > PetscInt
> >  > mypetsc[] in order to get the same ordering of the application
> >  > within PETSC?
> >  >
> >  >
> >  > Are you asking if the identity permutation can be constructed
> > using the
> >  > same array twice? Yes.
> >  >
> >  > And once I define the ordering so that the local vector and
> > matrix are
> >  > defined in PETSC as in my application, how can I use it to
> > create the
> >  > actual vector and matrix?
> >  >
> >  >
> >  > The vectors and matrices do not change. The AO is a permutation.
> > You can
> >  > use it to permute
> >  > a vector into another order, or to convert on index to another.
> >  >
> >  >Thanks,
> >  >
> >  >Matt
> >  >
> >  > Thanks in advance for the help.
> >  >
> >  > Cheers,
> >  > Enrico
> >  >
> >  > On 18/10/2023 13:39, Matthew Knepley wrote:
> >  >  > On Wed, Oct 18, 2023 at 5:55 AM Enrico  > <mailto:degreg...@dkrz.de>
> >  > <mailto:degreg...@dkrz.de <mailto:degreg...@dkrz.de>>
> &g

Re: [petsc-users] Coordinate format internal reordering

[Matthew Knepley]

On Thu, Oct 19, 2023 at 8:57 AM Enrico  wrote:

> Maybe I wasn't clear enough. I would like to completely get rid of Petsc
> ordering because I don't want extra communication between processes to
> construct the vector and the matrix (since I have to fill them every
> time step because I'm just using the linear solver with a Mat and a Vec
> data structure). I don't understand how I can do that.
>

Any program you write to do linear algebra will have contiguous storage
because it
is so much faster. Contiguous indexing makes sense for contiguous storage.
If you
want to use non-contiguous indexing for contiguous storage, you would need
some
translation layer. The AO is such a translation, but you could do this any
way you want.

  Thanks,

 Matt


> My initial idea was to create another global index ordering within my
> application to use only for the Petsc interface but then I think that
> the ghost cells are wrong.
>
> On 19/10/2023 14:50, Matthew Knepley wrote:
> > On Thu, Oct 19, 2023 at 6:51 AM Enrico  > <mailto:degreg...@dkrz.de>> wrote:
> >
> > Hello,
> >
> > if I create an application ordering using AOCreateBasic, should I
> > provide the same array for const PetscInt myapp[] and const PetscInt
> > mypetsc[] in order to get the same ordering of the application
> > within PETSC?
> >
> >
> > Are you asking if the identity permutation can be constructed using the
> > same array twice? Yes.
> >
> > And once I define the ordering so that the local vector and matrix
> are
> > defined in PETSC as in my application, how can I use it to create the
> > actual vector and matrix?
> >
> >
> > The vectors and matrices do not change. The AO is a permutation. You can
> > use it to permute
> > a vector into another order, or to convert on index to another.
> >
> >Thanks,
> >
> >Matt
> >
> > Thanks in advance for the help.
> >
> > Cheers,
> > Enrico
> >
> > On 18/10/2023 13:39, Matthew Knepley wrote:
> >  > On Wed, Oct 18, 2023 at 5:55 AM Enrico  > <mailto:degreg...@dkrz.de>
> >  > <mailto:degreg...@dkrz.de <mailto:degreg...@dkrz.de>>> wrote:
> >  >
> >  > Hello,
> >  >
> >  > I'm trying to use Petsc to solve a linear system in an
> > application. I'm
> >  > using the coordinate format to define the matrix and the
> > vector (it
> >  > should work better on GPU but at the moment every test is on
> > CPU).
> >  > After
> >  > the call to VecSetValuesCOO, I've noticed that the vector is
> > storing
> >  > the
> >  > data in a different way from my application. For example with
> two
> >  > processes in the application
> >  >
> >  > process 0 owns cells 2, 3, 4
> >  >
> >  > process 1 owns cells 0, 1, 5
> >  >
> >  > But in the vector data structure of Petsc
> >  >
> >  > process 0 owns cells 0, 1, 2
> >  >
> >  > process 1 owns cells 3, 4, 5
> >  >
> >  > This is in principle not a big issue, but after solving the
> > linear
> >  > system I get the solution vector x and I want to get the
> > values in the
> >  > correct processes. Is there a way to get vector values from
> other
> >  > processes or to get a mapping so that I can do it myself?
> >  >
> >  >
> >  > By definition, PETSc vectors and matrices own contiguous row
> > blocks. If
> >  > you want to have another,
> >  > global ordering, we support that with
> >  > https://petsc.org/main/manualpages/AO/
> > <https://petsc.org/main/manualpages/AO/>
> >  > <https://petsc.org/main/manualpages/AO/
> > <https://petsc.org/main/manualpages/AO/>>
> >  >
> >  >Thanks,
> >  >
> >  >   Matt
> >  >
> >  > Cheers,
> >  > Enrico Degregori
> >  >
> >  >
> >  >
> >  > --
> >  > What most experimenters take for granted before they begin their
> >  > experiments is infinitely more interesting than any results to
> which
> >  > their experiments lead.
> >  > -- Norbert Wiener
> >  >
> >  > https://www.cse.buffalo.edu/~knepley/
> > <https://www.cse.buffalo.edu/~knepley/>
> > <http://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/~knepley/>>
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ <
> http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Coordinate format internal reordering

[Matthew Knepley]

On Thu, Oct 19, 2023 at 6:51 AM Enrico  wrote:

> Hello,
>
> if I create an application ordering using AOCreateBasic, should I
> provide the same array for const PetscInt myapp[] and const PetscInt
> mypetsc[] in order to get the same ordering of the application within
> PETSC?
>

Are you asking if the identity permutation can be constructed using the
same array twice? Yes.


> And once I define the ordering so that the local vector and matrix are
> defined in PETSC as in my application, how can I use it to create the
> actual vector and matrix?
>

The vectors and matrices do not change. The AO is a permutation. You can
use it to permute
a vector into another order, or to convert on index to another.

  Thanks,

  Matt


> Thanks in advance for the help.
>
> Cheers,
> Enrico
>
> On 18/10/2023 13:39, Matthew Knepley wrote:
> > On Wed, Oct 18, 2023 at 5:55 AM Enrico  > <mailto:degreg...@dkrz.de>> wrote:
> >
> > Hello,
> >
> > I'm trying to use Petsc to solve a linear system in an application.
> I'm
> > using the coordinate format to define the matrix and the vector (it
> > should work better on GPU but at the moment every test is on CPU).
> > After
> > the call to VecSetValuesCOO, I've noticed that the vector is storing
> > the
> > data in a different way from my application. For example with two
> > processes in the application
> >
> > process 0 owns cells 2, 3, 4
> >
> > process 1 owns cells 0, 1, 5
> >
> > But in the vector data structure of Petsc
> >
> > process 0 owns cells 0, 1, 2
> >
> > process 1 owns cells 3, 4, 5
> >
> > This is in principle not a big issue, but after solving the linear
> > system I get the solution vector x and I want to get the values in
> the
> > correct processes. Is there a way to get vector values from other
> > processes or to get a mapping so that I can do it myself?
> >
> >
> > By definition, PETSc vectors and matrices own contiguous row blocks. If
> > you want to have another,
> > global ordering, we support that with
> > https://petsc.org/main/manualpages/AO/
> > <https://petsc.org/main/manualpages/AO/>
> >
> >Thanks,
> >
> >   Matt
> >
> > Cheers,
> > Enrico Degregori
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ <
> http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Error when installing PETSc

[Matthew Knepley]

On Wed, Oct 18, 2023 at 6:07 AM Gong Yujie 
wrote:

> Dear PETSc developers,
>
> I got an error message when installing PETSc with a clang compiler. Could
> you please help me find the problem? The configure.log is attached.
>

Your compiler segfaulted when compiling OpenMPI:

Making all in mca/crs
make[2]: Entering directory
'/home/tt/petsc-3.16.0/optamd/externalpackages/openmpi-4.1.0/opal/mca/crs'
  GENERATE opal_crs.7
  CC   base/crs_base_open.lo
  CC   base/crs_base_close.lo
  CC   base/crs_base_select.lo
  CC   base/crs_base_fns.lo
make[2]: Leaving directory
'/home/tt/petsc-3.16.0/optamd/externalpackages/openmpi-4.1.0/opal/mca/crs'
make[1]: Leaving directory
'/home/tt/petsc-3.16.0/optamd/externalpackages/openmpi-4.1.0/opal'/bin/sh:
line 7:  6327 Illegal instruction (core dumped) ../../../config/
make_manpage.pl --package-name='Open MPI' --package-version='4.1.0'
--ompi-date='Dec 18, 2020' --opal-date='Dec 18, 2020' --orte-date='Dec 18,
2020' --input=opal_crs.7in --output=opal_crs.7
make[2]: *** [Makefile:2215: opal_crs.7] Error 132
make[2]: *** Waiting for unfinished jobs
make[1]: *** [Makefile:2383: all-recursive] Error 1
make: *** [Makefile:1901: all-recursive] Error 1

I suggest compiling MPICH instead.

  Thanks,

 Matt


> Best Regards,
> Yujie
>
> Here is the detail of the error:
>
> =
>
>   Configuring PETSc to compile on your system
>
> =
> =
>
> * WARNING: Using default optimization C flags -g -O3
> You might consider manually setting optimal optimization flags for your
> system with
> COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for
> examples
> =
>
> =
>
> * WARNING: Using default Cxx optimization flags -g -O3
>
> You might consider manually setting optimal optimization flags for your
> system with
> CXXOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for
> examples
> =
>
> =
>
> * WARNING: Using default FORTRAN optimization flags -O
>
> You might consider manually setting optimal optimization flags for your
> system with
> FOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for
> examples
> =
>
> =
>
> Trying to download
> https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.0.tar.gz
> for
> =
>
> =
>
> Running configure on OPENMPI; this may take several minutes
>
> =
>
> =
>
> Running make on OPENMPI; this may take several minutes
>
> =
>
>
> ***
>  UNABLE to CONFIGURE with GIVEN OPTIONS(see configure.log for
> details):
>
> ---
> Error running make; make install on OPENMPI
>
> ***
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Coordinate format internal reordering

[Matthew Knepley]

On Wed, Oct 18, 2023 at 5:55 AM Enrico  wrote:

> Hello,
>
> I'm trying to use Petsc to solve a linear system in an application. I'm
> using the coordinate format to define the matrix and the vector (it
> should work better on GPU but at the moment every test is on CPU). After
> the call to VecSetValuesCOO, I've noticed that the vector is storing the
> data in a different way from my application. For example with two
> processes in the application
>
> process 0 owns cells 2, 3, 4
>
> process 1 owns cells 0, 1, 5
>
> But in the vector data structure of Petsc
>
> process 0 owns cells 0, 1, 2
>
> process 1 owns cells 3, 4, 5
>
> This is in principle not a big issue, but after solving the linear
> system I get the solution vector x and I want to get the values in the
> correct processes. Is there a way to get vector values from other
> processes or to get a mapping so that I can do it myself?
>

By definition, PETSc vectors and matrices own contiguous row blocks. If you
want to have another,
global ordering, we support that with https://petsc.org/main/manualpages/AO/

  Thanks,

 Matt


> Cheers,
> Enrico Degregori
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Using Sundials from PETSc

[Matthew Knepley]

On Mon, Oct 16, 2023 at 4:08 PM Vanella, Marcos (Fed) <
marcos.vane...@nist.gov> wrote:

> Hi Mathew, we have code that time splits the combustion step from the
> chemical species transport, so on each computational cell for each fluid
> flow time step, once transport is done we have the mixture chemical
> composition as initial condition. We are looking into doing finite rate
> chemistry with skeletal combustion models (20+ equations) in each cell for
> each fluid time step. Sundials provides the CVODE solver for the time
> integration of these, and would be interesting to see if we can make use of
> GPU acceleration. From their User Guide for Version 6.6.0 there are several
> GPU implementations for building RHS and using linear, nonlinear and stiff
> ODE solvers.
>

We are doing a similar thing in CHREST (https://www.buffalo.edu/chrest.html).
Since we normally use hundreds of species and thousands of reactions for
the reduced mechanism, we are using TChem2 to build and solve the system in
each cell.

Since these systems are so small, you are likely to need some way of
batching them within a warp. Do you have an idea for this already?

  Thanks,

 Matt


> Thank you Satish for the comment. Might be better at this point to first
> get an idea on what the implementation in our code using Sundials directly
> would look like. Then, we can see if it is possible and makes sense to
> access it through PETSc.
> We have things working in CPU making use of and older version of CVODE.
>
> BTW after some changes in our code we are starting running larger cases
> using GPU accelerated iterative solvers from PETSc, so we have PETSc
> interfaced already.
>
> Thanks!
>
> --
> *From:* Matthew Knepley 
> *Sent:* Monday, October 16, 2023 3:03 PM
> *To:* Vanella, Marcos (Fed) 
> *Cc:* petsc-users@mcs.anl.gov ; Paul, Chandan
> (IntlAssoc) 
> *Subject:* Re: [petsc-users] Using Sundials from PETSc
>
> On Mon, Oct 16, 2023 at 2:29 PM Vanella, Marcos (Fed) via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Hi, we were wondering if it would be possible to call the latest version
> of Sundials from PETSc?
>
>
> The short answer is, no. We are at v2.5 and they are at v6.5. There were
> no dates on the version history page, so I do not know how out of date we
> are. There have not been any requests for update until now.
>
> We would be happy to get an MR for the updates if you want to try it.
>
>
> We are interested in doing chemistry using GPUs and already have
> interfaces to PETSc from our code.
>
>
> How does the GPU interest interact with the SUNDIALS version?
>
>   Thanks,
>
>  Matt
>
>
> Thanks,
> Marcos
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Using Sundials from PETSc

[Matthew Knepley]

On Mon, Oct 16, 2023 at 2:29 PM Vanella, Marcos (Fed) via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi, we were wondering if it would be possible to call the latest version
> of Sundials from PETSc?
>

The short answer is, no. We are at v2.5 and they are at v6.5. There were no
dates on the version history page, so I do not know how out of date we are.
There have not been any requests for update until now.

We would be happy to get an MR for the updates if you want to try it.


> We are interested in doing chemistry using GPUs and already have
> interfaces to PETSc from our code.
>

How does the GPU interest interact with the SUNDIALS version?

  Thanks,

 Matt


> Thanks,
> Marcos
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Parallel DMPlex

[Matthew Knepley]

On Mon, Oct 16, 2023 at 10:10 AM erdemguer  wrote:

> I'm truly sorry for my bad.
> I set the nDof = 1 for simplicity. You can find my code in the
> attachments. In that code I tried to find an example of a cell which is
> neighbor to a cell in the another processor and print them.
> Here is my output:
> (base) ➜  build git:(main) ✗ /petsc/lib/petsc/bin/petscmpiexec -n 2
> ./ex1_eg  -dm_plex_dim 3 -dm_plex_simplex 0 -dm_plex_box_faces 3,3,3
> Before Distribution Rank: 0, cStart: 0, cEndInterior: 27, cEnd: 27
> Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
> After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 27
> After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 24
> [0] m: 13 n: 13
> [1] m: 14 n: 14
> [1] Face: 94, Center Cell: 7, Ghost Neighbor Cell: 23
> [0] Face: 145, Center Cell: 12, Ghost Neighbor Cell: 20
>

You can force us to have the same partition using  -petscpartitioner_type
simple,

master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1
-malloc_debug 0 -dm_plex_dim 3 -dm_plex_simplex 0 -dm_plex_box_faces 3,3,3
-petscpartitioner_type simple
Before Distribution Rank: 0, cStart: 0, cEndInterior: 27, cEnd: 27
Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
After Distribution Rank: 0, cStart: 0, cEndInterior: 14, cEnd: 27
After Distribution Rank: 1, cStart: 0, cEndInterior: 13, cEnd: 26
[0] m: 14 n: 14
[1] m: 13 n: 13
[1] Face: 89, Center Cell: 0, Ghost Neighbor Cell: 25
[0] Face: 140, Center Cell: 13, Ghost Neighbor Cell: 14


> For example, if I'm writing residual for cell 12 on rank 0, I thought I
> need to write on (12,20) on the matrix too. But looks like that isn't the
> case.
>

There are two problems here:

1) 20 is the _local_ number of that cell, but matrices use global numbers.
If you want to know that global number of that dof,
it is two steps. First, you need the cell number on the other process. You
can get this from the pointSF. If

   leaves[i] = 20, then remotes[i].index = 

Then you need the dof for that remote cell. However, this work has already
been done by the global Section. So

  DMGetGlobalSection(dm, &gsec);
  PetscSectionGetOffset(gsec, 20, &off);
  off = -(off + 1);

since dofs we do not own will be encoded as -(dof + 1).

2) You need to decide how you want to assemble. Do we assemble the
contributions from the cells we own, or from the faces we own. Most FV
people divide up the faces.

  Thanks,

 Matt


> Thanks,
> Guer
>
> Sent with Proton Mail <https://proton.me/> secure email.
>
> --- Original Message ---
> On Monday, October 16th, 2023 at 4:26 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Mon, Oct 16, 2023 at 9:22 AM erdemguer  wrote:
>
>> Thank you for your responses many times. Looks like I'm missing
>> something, sorry for my confusion, but let's take processor 0 on your first
>> output. cEndInterior: 16 and cEnd: 24.
>> I'm calculating jacobian for cell=14, dof=0 (row = 42) and cell=18, dof=2
>> (col = 56) have influence on it. (Cell 18 is on processor boundary)
>> Shouldn't I have to write values on the (42,56)?
>>
>
> Imagine you are me getting this mail. When I mail you, I show you
> _exactly_ what I ran and which command line options I used. You do not. I
> provide you all the output. You do not. You can see that someone would only
> be guessing when replying to this email. Also note that you have two dofs
> per cell, so the cell numbers are not the row numbers for the Jacobian.
> Please send something reproducible when you want help on running.
>
> Thanks,
>
> Matt
>
>> Thanks,
>> Guer
>>
>> Sent with Proton Mail <https://proton.me/> secure email.
>>
>> --- Original Message ---
>> On Monday, October 16th, 2023 at 4:11 PM, Matthew Knepley <
>> knep...@gmail.com> wrote:
>>
>> On Mon, Oct 16, 2023 at 6:54 AM erdemguer  wrote:
>>
>>> Hey again.
>>>
>>> This code outputs for example:
>>>
>>> After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 24
>>> After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 27
>>> [0] m: 39 n: 39
>>> [1] m: 42 n: 42
>>>
>>> Shouldn't it be 39 x 81 and 42 x 72 because of the overlapping cells on
>>> processor boundaries?
>>>
>>
>> Here is my output
>>
>> master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1
>> -malloc_debug 0 -dm_refine 1
>> Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
>> Before Distribution Rank: 0, cStart: 0, cEndInterior: 32, cEnd: 32
>> After Distribution Rank: 1, cStart: 0, cEndInt

Re: [petsc-users] Parallel DMPlex

[Matthew Knepley]

On Mon, Oct 16, 2023 at 9:22 AM erdemguer  wrote:

> Thank you for your responses many times. Looks like I'm missing something,
> sorry for my confusion, but let's take processor 0 on your first output.
> cEndInterior: 16 and cEnd: 24.
> I'm calculating jacobian for cell=14, dof=0 (row = 42) and cell=18, dof=2
> (col = 56) have influence on it. (Cell 18 is on processor boundary)
> Shouldn't I have to write values on the (42,56)?
>

Imagine you are me getting this mail. When I mail you, I show you _exactly_
what I ran and which command line  options I used. You do not. I provide
you all the output. You do not. You can see that someone would only be
guessing when replying to this email. Also note that you have two dofs per
cell, so the cell numbers are not the row numbers for the Jacobian. Please
send something reproducible when you want help on running.

  Thanks,

 Matt


> Thanks,
> Guer
>
> Sent with Proton Mail <https://proton.me/> secure email.
>
> --- Original Message ---
> On Monday, October 16th, 2023 at 4:11 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Mon, Oct 16, 2023 at 6:54 AM erdemguer  wrote:
>
>> Hey again.
>>
>> This code outputs for example:
>>
>> After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 24
>> After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 27
>> [0] m: 39 n: 39
>> [1] m: 42 n: 42
>>
>> Shouldn't it be 39 x 81 and 42 x 72 because of the overlapping cells on
>> processor boundaries?
>>
>
> Here is my output
>
> master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1
> -malloc_debug 0 -dm_refine 1
> Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
> Before Distribution Rank: 0, cStart: 0, cEndInterior: 32, cEnd: 32
> After Distribution Rank: 1, cStart: 0, cEndInterior: 16, cEnd: 24
> After Distribution Rank: 0, cStart: 0, cEndInterior: 16, cEnd: 24
> [0] m: 48 n: 48
> [1] m: 48 n: 48
>
> The mesh is 4x4 and also split into two triangles, so 32 triangles. Then
> we split it and have 8 overlap cells on each side. You can get quads using
>
> master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1
> -malloc_debug 0 -dm_plex_simplex 0 -dm_refine 1 -dm_view
> Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
> Before Distribution Rank: 0, cStart: 0, cEndInterior: 16, cEnd: 16
> After Distribution Rank: 1, cStart: 0, cEndInterior: 8, cEnd: 12
> After Distribution Rank: 0, cStart: 0, cEndInterior: 8, cEnd: 12
> [0] m: 24 n: 24
> [1] m: 24 n: 24
> It is the same 4x4 mesh, but now with quads.
>
> Thanks,
>
> Matt
>
> P.S. It looks like I should use PetscFV or something like that at the
>> first place. At first I thought, "I will just use SNES, I will compute only
>> residual and jacobian on cells so why do bother with PetscFV?" So
>>
>> Thanks,
>> E.
>> Sent with Proton Mail <https://proton.me/> secure email.
>>
>> --- Original Message ---
>> On Friday, October 13th, 2023 at 3:00 PM, Matthew Knepley <
>> knep...@gmail.com> wrote:
>>
>> On Fri, Oct 13, 2023 at 7:26 AM erdemguer  wrote:
>>
>>> Hi, unfortunately it's me again.
>>>
>>> I have some weird troubles with creating matrix with DMPlex. Actually I
>>> might not need to create matrix explicitly, but SNESSolve crashes at there
>>> too. So, I updated the code you provided. When I tried to use
>>> DMCreateMatrix() at first, I got an error "Unknown discretization type
>>> for field 0" at first I applied DMSetLocalSection() and this error is gone.
>>> But this time when I run the code with multiple processors, sometimes I got
>>> an output like:
>>>
>>
>> Some setup was out of order so the section size on proc1 was 0, and I was
>> not good about checking this.
>> I have fixed it and attached.
>>
>> Thanks,
>>
>> Matt
>>
>> Before Distribution Rank: 0, cStart: 0, cEndInterior: 27, cEnd: 27
>>> Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
>>> [1] ghost cell 14
>>> [1] ghost cell 15
>>> [1] ghost cell 16
>>> [1] ghost cell 17
>>> [1] ghost cell 18
>>> [1] ghost cell 19
>>> [1] ghost cell 20
>>> [1] ghost cell 21
>>> [1] ghost cell 22
>>> After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 23
>>> [0] ghost cell 13
>>> [0] ghost cell 14
>>> [0] ghost cell 15
>>> [0] ghost cell 16
>>> [0] ghost cell 17
>>> [0] g

Re: [petsc-users] Parallel DMPlex

[Matthew Knepley]

On Mon, Oct 16, 2023 at 6:54 AM erdemguer  wrote:

> Hey again.
>
> This code outputs for example:
>
> After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 24
> After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 27
> [0] m: 39 n: 39
> [1] m: 42 n: 42
>
> Shouldn't it be 39 x 81 and 42 x 72 because of the overlapping cells on
> processor boundaries?
>

Here is my output

master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1
-malloc_debug 0 -dm_refine 1
Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
Before Distribution Rank: 0, cStart: 0, cEndInterior: 32, cEnd: 32
After Distribution Rank: 1, cStart: 0, cEndInterior: 16, cEnd: 24
After Distribution Rank: 0, cStart: 0, cEndInterior: 16, cEnd: 24
[0] m: 48 n: 48
[1] m: 48 n: 48

The mesh is 4x4 and also split into two triangles, so 32 triangles. Then we
split it and have 8 overlap cells on each side. You can get quads using

master *:~/Downloads/tmp/Guer$ /PETSc3/petsc/apple/bin/mpiexec -n 2 ./ex1
-malloc_debug 0 -dm_plex_simplex 0 -dm_refine 1 -dm_view
Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
Before Distribution Rank: 0, cStart: 0, cEndInterior: 16, cEnd: 16
After Distribution Rank: 1, cStart: 0, cEndInterior: 8, cEnd: 12
After Distribution Rank: 0, cStart: 0, cEndInterior: 8, cEnd: 12
[0] m: 24 n: 24
[1] m: 24 n: 24

It is the same 4x4 mesh, but now with quads.

  Thanks,

 Matt

P.S. It looks like I should use PetscFV or something like that at the first
> place. At first I thought, "I will just use SNES, I will compute only
> residual and jacobian on cells so why do bother with PetscFV?" So
>
> Thanks,
> E.
> Sent with Proton Mail <https://proton.me/> secure email.
>
> ------- Original Message ---
> On Friday, October 13th, 2023 at 3:00 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Fri, Oct 13, 2023 at 7:26 AM erdemguer  wrote:
>
>> Hi, unfortunately it's me again.
>>
>> I have some weird troubles with creating matrix with DMPlex. Actually I
>> might not need to create matrix explicitly, but SNESSolve crashes at there
>> too. So, I updated the code you provided. When I tried to use
>> DMCreateMatrix() at first, I got an error "Unknown discretization type
>> for field 0" at first I applied DMSetLocalSection() and this error is gone.
>> But this time when I run the code with multiple processors, sometimes I got
>> an output like:
>>
>
> Some setup was out of order so the section size on proc1 was 0, and I was
> not good about checking this.
> I have fixed it and attached.
>
> Thanks,
>
> Matt
>
> Before Distribution Rank: 0, cStart: 0, cEndInterior: 27, cEnd: 27
>> Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
>> [1] ghost cell 14
>> [1] ghost cell 15
>> [1] ghost cell 16
>> [1] ghost cell 17
>> [1] ghost cell 18
>> [1] ghost cell 19
>> [1] ghost cell 20
>> [1] ghost cell 21
>> [1] ghost cell 22
>> After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 23
>> [0] ghost cell 13
>> [0] ghost cell 14
>> [0] ghost cell 15
>> [0] ghost cell 16
>> [0] ghost cell 17
>> [0] ghost cell 18
>> [0] ghost cell 19
>> [0] ghost cell 20
>> [0] ghost cell 21
>> [0] ghost cell 22
>> [0] ghost cell 23
>> After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 24
>> Fatal error in internal_Waitall: Unknown error class, error stack:
>> internal_Waitall(82)..: MPI_Waitall(count=1,
>> array_of_requests=0xf5f72264, array_of_statuses=0x1) failed
>> MPIR_Waitall(1099):
>> MPIR_Waitall_impl(1011)...:
>> MPIR_Waitall_state(976)...:
>> MPIDI_CH3i_Progress_wait(187).: an error occurred while
>> handling an event returned by MPIDI_CH3I_Sock_Wait()
>> MPIDI_CH3I_Progress_handle_sock_event(411):
>> ReadMoreData(744).: ch3|sock|immedread
>> 0x8851c5c0 0xf5e81cd0 0xf5e8a880
>> MPIDI_CH3I_Sock_readv(2553)...: the supplied buffer contains
>> invalid memory (set=0,sock=1,errno=14:Bad address)
>>
>> Sometimes the error message isn't appearing but for example I'm trying to
>> print size of the matrix but it isn't working.
>> If necessary, my Configure options --download-mpich --download-hwloc
>> --download-pastix --download-hypre --download-ml --download-ctetgen
>> --download-triangle --download-exodusii --download-netcdf --download-zlib
>> --download-pnetcdf --download-ptscotch --download-hdf5 --with-cc=clang-16
>> --with-cxx=clang++-16 COPTFLAGS

Re: [petsc-users] Parallel DMPlex

[Matthew Knepley]

On Fri, Oct 13, 2023 at 7:26 AM erdemguer  wrote:

> Hi, unfortunately it's me again.
>
> I have some weird troubles with creating matrix with DMPlex. Actually I
> might not need to create matrix explicitly, but SNESSolve crashes at there
> too. So, I updated the code you provided. When I tried to use
> DMCreateMatrix() at first, I got an error "Unknown discretization type
> for field 0" at first I applied DMSetLocalSection() and this error is gone.
> But this time when I run the code with multiple processors, sometimes I got
> an output like:
>

Some setup was out of order so the section size on proc1 was 0, and I was
not good about checking this.
I have fixed it and attached.

  Thanks,

 Matt

Before Distribution Rank: 0, cStart: 0, cEndInterior: 27, cEnd: 27
> Before Distribution Rank: 1, cStart: 0, cEndInterior: 0, cEnd: 0
> [1] ghost cell 14
> [1] ghost cell 15
> [1] ghost cell 16
> [1] ghost cell 17
> [1] ghost cell 18
> [1] ghost cell 19
> [1] ghost cell 20
> [1] ghost cell 21
> [1] ghost cell 22
> After Distribution Rank: 1, cStart: 0, cEndInterior: 14, cEnd: 23
> [0] ghost cell 13
> [0] ghost cell 14
> [0] ghost cell 15
> [0] ghost cell 16
> [0] ghost cell 17
> [0] ghost cell 18
> [0] ghost cell 19
> [0] ghost cell 20
> [0] ghost cell 21
> [0] ghost cell 22
> [0] ghost cell 23
> After Distribution Rank: 0, cStart: 0, cEndInterior: 13, cEnd: 24
> Fatal error in internal_Waitall: Unknown error class, error stack:
> internal_Waitall(82)..: MPI_Waitall(count=1,
> array_of_requests=0xf5f72264, array_of_statuses=0x1) failed
> MPIR_Waitall(1099):
> MPIR_Waitall_impl(1011)...:
> MPIR_Waitall_state(976)...:
> MPIDI_CH3i_Progress_wait(187).: an error occurred while
> handling an event returned by MPIDI_CH3I_Sock_Wait()
> MPIDI_CH3I_Progress_handle_sock_event(411):
> ReadMoreData(744).: ch3|sock|immedread
> 0x8851c5c0 0xf5e81cd0 0xf5e8a880
> MPIDI_CH3I_Sock_readv(2553)...: the supplied buffer contains
> invalid memory (set=0,sock=1,errno=14:Bad address)
>
> Sometimes the error message isn't appearing but for example I'm trying to
> print size of the matrix but it isn't working.
> If necessary, my Configure options --download-mpich --download-hwloc
> --download-pastix --download-hypre --download-ml --download-ctetgen
> --download-triangle --download-exodusii --download-netcdf --download-zlib
> --download-pnetcdf --download-ptscotch --download-hdf5 --with-cc=clang-16
> --with-cxx=clang++-16 COPTFLAGS="-g -O2" CXXOPTFLAGS="-g -O2" FOPTFLAGS="-g
> -O2" --with-debugging=1
>
> Version: Petsc Release Version 3.20.0
>
> Thank you,
> Guer
>
> Sent with Proton Mail <https://proton.me/> secure email.
>
> --- Original Message ---
> On Thursday, October 12th, 2023 at 12:59 AM, erdemguer <
> erdemg...@proton.me> wrote:
>
> Thank you! That's exactly what I need.
>
> Sent with Proton Mail <https://proton.me/> secure email.
>
> --- Original Message ---
> On Wednesday, October 11th, 2023 at 4:17 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Wed, Oct 11, 2023 at 4:42 AM erdemguer  wrote:
>
>> Hi again,
>>
>
> I see the problem. FV ghosts mean extra boundary cells added in FV methods
> using DMPlexCreateGhostCells() in order to impose boundary conditions. They
> are not the "ghost" cells for overlapping parallel decompositions. I have
> changed your code to give you what you want. It is attached.
>
> Thanks,
>
> Matt
>
>> Here is my code:
>> #include 
>> static char help[] = "dmplex";
>>
>> int main(int argc, char **argv)
>> {
>> PetscCall(PetscInitialize(&argc, &argv, NULL, help));
>> DM dm, dm_dist;
>> PetscSection section;
>> PetscInt cStart, cEndInterior, cEnd, rank;
>> PetscInt nc[3] = {3, 3, 3};
>> PetscReal upper[3] = {1, 1, 1};
>>
>> PetscCallMPI(MPI_Comm_rank(PETSC_COMM_WORLD, &rank));
>>
>> DMPlexCreateBoxMesh(PETSC_COMM_WORLD, 3, PETSC_FALSE, nc, NULL, upper,
>> NULL, PETSC_TRUE, &dm);
>> DMViewFromOptions(dm, NULL, "-dm1_view");
>> PetscCall(DMSetFromOptions(dm));
>> DMViewFromOptions(dm, NULL, "-dm2_view");
>>
>> PetscCall(DMPlexGetDepthStratum(dm, 3, &cStart, &cEnd));
>> DMPlexComputeCellTypes(dm);
>> PetscCall(DMPlexGetCellTypeStratum(dm, DM_POLYTOPE_INTERIOR_GHOST,
>> &cEndInterior, NULL));
>> PetscPrintf(PETSC_COMM_SELF, "Before Distribution Rank: %d

Re: [petsc-users] FEM Implementation of NS with SUPG Stabilization

[Matthew Knepley]

On Wed, Oct 11, 2023 at 4:15 PM Brandon Denton  wrote:

> By natural coordinates, I am referring to the reference element
> coordinates. Usually these are represented as (xi, eta, zeta) in the
> literature.
>
> Yes. I would like to have the Jacobian and the derivatives of the map
> available within PetscDSSetResidual() f0 and f1 functions.
>

Yes, we can get these passed an aux data.


>   I believe DMPlexComputeCellGeometryFEM() function provides this
> information. Is there a way to get the cell, shape functions as well? It
> not, can we talk about this more? I would like to understand how the shape
> functions are addressed within PETSc. Dr. Kirk's approach uses the shape
> function gradients in its SUPG parameter. I'd love to talk with you about
> this is more detail.
>

There should be a way to formulate this in a basis independent way.  I
would much prefer that to
explicit inclusion of the basis.

  Thanks,

 Matt


> *From:* Matthew Knepley 
> *Sent:* Wednesday, October 11, 2023 3:13 PM
> *To:* Brandon Denton 
> *Cc:* Jed Brown ; petsc-users 
> *Subject:* Re: [petsc-users] FEM Implementation of NS with SUPG
> Stabilization
>
> On Wed, Oct 11, 2023 at 2:09 PM Brandon Denton 
> wrote:
>
> Thank you for the discussion.
>
> Are we agreed then that the derivatives of the natural coordinates are
> required for the described approach? If so, is this something PETSc can
> currently do within the point-wise residual functions?
>
>
> I am not sure what natural coordinates are. Do we just mean the Jacobian,
> derivatives of the map between reference and real coordinates? If so, yes
> the Jacobian is available. Right now I do not pass it
> directly, but passing it is easy.
>
>   Thanks,
>
>  Matt
>
>
> Matt - Thank you for the command line option for the 2nd derivatives.
> Those will be needed to implement the discussed approach. Specifically in
> the stabilization and shock capture parameters. (Ref.: B. Kirk's Thesis).
> What is a good reference for the usual SUPG method you are referencing?
> I've been looking through my textbooks but haven't found a good reference.
>
> Jed - Thank you for the link. I will review the information on it.
>
> Sorry about the attachment. I will upload it to this thread later (I'm at
> work right now and I can't do it from here).
> --
> *From:* Jed Brown 
> *Sent:* Wednesday, October 11, 2023 1:38 PM
> *To:* Matthew Knepley 
> *Cc:* Brandon Denton ; petsc-users <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] FEM Implementation of NS with SUPG
> Stabilization
>
> Matthew Knepley  writes:
>
> > On Wed, Oct 11, 2023 at 1:03 PM Jed Brown  wrote:
> >
> >> I don't see an attachment, but his thesis used conservative variables
> and
> >> defined an effective length scale in a way that seemed to assume
> constant
> >> shape function gradients. I'm not aware of systematic literature
> comparing
> >> the covariant and contravariant length measures on anisotropic meshes,
> but
> >> I believe most people working in the Shakib/Hughes approach use the
> >> covariant measure. Our docs have a brief discussion of this choice.
> >>
> >>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flibceed.org%2Fen%2Flatest%2Fexamples%2Ffluids%2F%23equation-eq-peclet&data=05%7C01%7Cbldenton%40buffalo.edu%7Cd9372f934b26455371a708dbca80dc8e%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C638326427028053956%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=skMsKDmpBxiaXtBSqhsyckvVpTOkGqDsNJIYo22Ywps%3D&reserved=0
> <https://libceed.org/en/latest/examples/fluids/#equation-eq-peclet>
> >>
> >> Matt, I don't understand how the second derivative comes into play as a
> >> length measure on anistropic meshes -- the second derivatives can be
> >> uniformly zero and yet you still need a length measure.
> >>
> >
> > I was talking about the usual SUPG where we just penalize the true
> residual.
>
> I think you're focused on computing the strong diffusive flux (which can
> be done using second derivatives or by a projection; the latter produces
> somewhat better results). But you still need a length scale and that's most
> naturally computed using the derivative of reference coordinates with
> respect to physical (or equivalently, the associated metric tensor).
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] FEM Implementation of NS with SUPG Stabilization

[Matthew Knepley]

On Wed, Oct 11, 2023 at 2:09 PM Brandon Denton  wrote:

> Thank you for the discussion.
>
> Are we agreed then that the derivatives of the natural coordinates are
> required for the described approach? If so, is this something PETSc can
> currently do within the point-wise residual functions?
>

I am not sure what natural coordinates are. Do we just mean the Jacobian,
derivatives of the map between reference and real coordinates? If so, yes
the Jacobian is available. Right now I do not pass it
directly, but passing it is easy.

  Thanks,

 Matt


> Matt - Thank you for the command line option for the 2nd derivatives.
> Those will be needed to implement the discussed approach. Specifically in
> the stabilization and shock capture parameters. (Ref.: B. Kirk's Thesis).
> What is a good reference for the usual SUPG method you are referencing?
> I've been looking through my textbooks but haven't found a good reference.
>
> Jed - Thank you for the link. I will review the information on it.
>
> Sorry about the attachment. I will upload it to this thread later (I'm at
> work right now and I can't do it from here).
> --
> *From:* Jed Brown 
> *Sent:* Wednesday, October 11, 2023 1:38 PM
> *To:* Matthew Knepley 
> *Cc:* Brandon Denton ; petsc-users <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] FEM Implementation of NS with SUPG
> Stabilization
>
> Matthew Knepley  writes:
>
> > On Wed, Oct 11, 2023 at 1:03 PM Jed Brown  wrote:
> >
> >> I don't see an attachment, but his thesis used conservative variables
> and
> >> defined an effective length scale in a way that seemed to assume
> constant
> >> shape function gradients. I'm not aware of systematic literature
> comparing
> >> the covariant and contravariant length measures on anisotropic meshes,
> but
> >> I believe most people working in the Shakib/Hughes approach use the
> >> covariant measure. Our docs have a brief discussion of this choice.
> >>
> >>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flibceed.org%2Fen%2Flatest%2Fexamples%2Ffluids%2F%23equation-eq-peclet&data=05%7C01%7Cbldenton%40buffalo.edu%7Cd9372f934b26455371a708dbca80dc8e%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C638326427028053956%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=skMsKDmpBxiaXtBSqhsyckvVpTOkGqDsNJIYo22Ywps%3D&reserved=0
> <https://libceed.org/en/latest/examples/fluids/#equation-eq-peclet>
> >>
> >> Matt, I don't understand how the second derivative comes into play as a
> >> length measure on anistropic meshes -- the second derivatives can be
> >> uniformly zero and yet you still need a length measure.
> >>
> >
> > I was talking about the usual SUPG where we just penalize the true
> residual.
>
> I think you're focused on computing the strong diffusive flux (which can
> be done using second derivatives or by a projection; the latter produces
> somewhat better results). But you still need a length scale and that's most
> naturally computed using the derivative of reference coordinates with
> respect to physical (or equivalently, the associated metric tensor).
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] FEM Implementation of NS with SUPG Stabilization

[Matthew Knepley]

On Wed, Oct 11, 2023 at 1:03 PM Jed Brown  wrote:

> I don't see an attachment, but his thesis used conservative variables and
> defined an effective length scale in a way that seemed to assume constant
> shape function gradients. I'm not aware of systematic literature comparing
> the covariant and contravariant length measures on anisotropic meshes, but
> I believe most people working in the Shakib/Hughes approach use the
> covariant measure. Our docs have a brief discussion of this choice.
>
> https://libceed.org/en/latest/examples/fluids/#equation-eq-peclet
>
> Matt, I don't understand how the second derivative comes into play as a
> length measure on anistropic meshes -- the second derivatives can be
> uniformly zero and yet you still need a length measure.
>

I was talking about the usual SUPG where we just penalize the true residual.

  Matt


> Brandon Denton via petsc-users  writes:
>
> > I was thinking about trying to implement Ben Kirk's approach to
> Navier-Stokes (see attached paper; Section 5). His approach uses these
> quantities to align the orientation of the unstructured element/cell with
> the fluid velocity to apply the stabilization/upwinding and to detect
> shocks.
> >
> > If you have an example of the approach you mentioned, could you please
> send it over so I can review it?
> >
> > On Oct 11, 2023 6:02 AM, Matthew Knepley  wrote:
> > On Tue, Oct 10, 2023 at 9:34 PM Brandon Denton via petsc-users <
> petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>> wrote:
> > Good Evening,
> >
> > I am looking to implement a form of Navier-Stokes with SUPG
> Stabilization and shock capturing using PETSc's FEM infrastructure. In this
> implementation, I need access to the cell's shape function gradients and
> natural coordinate gradients for calculations within the point-wise
> residual calculations. How do I get these quantities at the quadrature
> points? The signatures for fo and f1 don't seem to contain this information.
> >
> > Are you sure you need those? Darsh and I implemented SUPG without that.
> You would need local second derivative information, which you can get using
> -dm_ds_jet_degree 2. If you check in an example, I can go over it.
> >
> >   Thanks,
> >
> >  Matt
> >
> > Thank you in advance for your time.
> > Brandon
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/<
> http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Parallel DMPlex

[Matthew Knepley]

ue/size (1 (6), 2 (1), 3 (7), 5 (5), 6 (4))
>   celltype: 4 strata with value/size (0 (40), 1 (83), 4 (57), 7 (13))
> Field Field_0:
>   adjacency FEM
>
> dm4_view (after setting adjacency):
> DM Object: 2 MPI processes
>   type: plex
> DM_0x8404_0 in 3 dimensions:
>   Number of 0-cells per rank: 40 46
>   Number of 1-cells per rank: 83 95
>   Number of 2-cells per rank: 57 64
>   Number of 3-cells per rank: 13 14
> Labels:
>   depth: 4 strata with value/size (0 (40), 1 (83), 2 (57), 3 (13))
>   marker: 1 strata with value/size (1 (109))
>   Face Sets: 5 strata with value/size (1 (6), 2 (1), 3 (7), 5 (5), 6 (4))
>   celltype: 4 strata with value/size (0 (40), 1 (83), 4 (57), 7 (13))
> Field Field_0:
>   adjacency FVM++
>
> dm5_view (after distribution):
> DM Object: Parallel Mesh 2 MPI processes
>   type: plex
> Parallel Mesh in 3 dimensions:
>   Number of 0-cells per rank: 64 60
>   Number of 1-cells per rank: 144 133
>   Number of 2-cells per rank: 108 98
>   Number of 3-cells per rank: 27 24
> Labels:
>   depth: 4 strata with value/size (0 (64), 1 (144), 2 (108), 3 (27))
>   marker: 1 strata with value/size (1 (218))
>   Face Sets: 6 strata with value/size (1 (9), 2 (9), 3 (9), 4 (9), 5 (9),
> 6 (9))
>   celltype: 4 strata with value/size (0 (64), 1 (144), 4 (108), 7 (27))
> Field Field_0:
>   adjacency FVM++
>
> Thanks,
> Guer.
> Sent with Proton Mail <https://proton.me/> secure email.
>
> --- Original Message ---
> On Wednesday, October 11th, 2023 at 3:33 AM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Tue, Oct 10, 2023 at 7:01 PM erdemguer  wrote:
>
>>
>> Hi,
>> Sorry for my late response. I tried with your suggestions and I think I
>> made a progress. But I still got issues. Let me explain my latest mesh
>> routine:
>>
>>
>>1. DMPlexCreateBoxMesh
>>2. DMSetFromOptions
>>3. PetscSectionCreate
>>4. PetscSectionSetNumFields
>>5. PetscSectionSetFieldDof
>>6. PetscSectionSetDof
>>7. PetscSectionSetUp
>>8. DMSetLocalSection
>>9. DMSetAdjacency
>>10. DMPlexDistribute
>>
>>
>> It's still not working but it's promising, if I call
>> DMPlexGetDepthStratum for cells, I can see that after distribution
>> processors have more cells.
>>
>
> Please send the output of DMPlexView() for each incarnation of the mesh.
> What I do is put
>
> DMViewFromOptions(dm, NULL, "-dm1_view")
>
>
> with a different string after each call.
>
>> But I couldn't figure out how to decide where the ghost/processor
>> boundary cells start.
>>
>
> Please send the actual code because the above is not specific enough. For
> example, you will not have
> "ghost cells" unless you partition with overlap. This is because by
> default cells are the partitioned quantity,
> so each process gets a unique set.
>
> Thanks,
>
> Matt
>
>> In older mails I saw there is a function DMPlexGetHybridBounds but I
>> think that function is deprecated. I tried to use,
>> DMPlexGetCellTypeStratum as in ts/tutorials/ex11_sa.c but I'm getting -1
>> as cEndInterior before and after distribution. I tried it for 
>> DM_POLYTOPE_FV_GHOST,
>> DM_POLYTOPE_INTERIOR_GHOST polytope types. I also tried calling
>> DMPlexComputeCellTypes before DMPlexGetCellTypeStratum but nothing changed.
>> I think I can calculate the ghost cell indices using cStart/cEnd before &
>> after distribution but I think there is a better way I'm currently missing.
>>
>> Thanks again,
>> Guer.
>>
>> --- Original Message ---
>> On Thursday, September 28th, 2023 at 10:42 PM, Matthew Knepley <
>> knep...@gmail.com> wrote:
>>
>> On Thu, Sep 28, 2023 at 3:38 PM erdemguer via petsc-users <
>> petsc-users@mcs.anl.gov> wrote:
>>
>>> Hi,
>>>
>>> I am currently using DMPlex in my code. It runs serially at the moment,
>>> but I'm interested in adding parallel options. Here is my workflow:
>>>
>>> Create a DMPlex mesh from GMSH.
>>> Reorder it with DMPlexPermute.
>>> Create necessary pre-processing arrays related to the mesh/problem.
>>> Create field(s) with multi-dofs.
>>> Create residual vectors.
>>> Define a function to calculate the residual for each cell and, use SNES.
>>> As you can see, I'm not using FV or FE structures (most examples do).
>>> Now, I'm trying to implement this in parallel using a similar approach.
>>> However, I'm struggling to understand

Re: [petsc-users] Compilation failure of PETSc with "The procedure name of the INTERFACE block conflicts with a name in the encompassing scoping unit"

[Matthew Knepley]

On Wed, Oct 11, 2023 at 4:22 AM Richter, Roland 
wrote:

> Hei,
>
> following my last question I managed to configure PETSc with Intel MPI and
> Intel OneAPI using the following configure-line:
>
>
>
> *./configure --prefix=/media/storage/local_opt/petsc
> --with-scalar-type=complex --with-cc=mpiicc --with-cxx=mpiicpc
> --CPPFLAGS="-fPIC -march=native -mavx2" --CXXFLAGS="-fPIC -march=native
> -mavx2" --with-fc=mpiifort --with-pic=true --with-mpi=true
> --with-blaslapack-dir=/opt/intel/oneapi/mkl/latest/lib/intel64/
> --with-openmp=true --download-hdf5=yes --download-netcdf=yes
> --download-chaco=no --download-metis=yes --download-slepc=yes
> --download-suitesparse=yes --download-eigen=yes --download-parmetis=yes
> --download-ptscotch=yes --download-mumps=yes --download-scalapack=yes
> --download-superlu=yes --download-superlu_dist=yes --with-mkl_pardiso=1
> --with-boost=1 --with-boost-dir=/media/storage/local_opt/boost
> --download-opencascade=yes --with-fftw=1
> --with-fftw-dir=/media/storage/local_opt/fftw3 --download-kokkos=yes
> --with-mkl_sparse=1 --with-mkl_cpardiso=1 --with-mkl_sparse_optimize=1
> --download-muparser=yes --download-p4est=yes --download-sowing=yes
> --download-viennalcl=yes --with-zlib --force=1 --with-clean=1 --with-cuda=0*
>
>
>
> Now, however, compilation fails with the following error:
>
> /home/user/Downloads/git-files/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90(699):
> error #6623: The procedure name of the INTERFACE block conflicts with a
> name in the encompassing scoping unit.   [PCGASMCREATESUBDOMAINS2D]
>
>   subroutine PCGASMCreateSubdomains2D(a,b,c,d,e,f,g,h,i,j,z)
>
> -^
>
> /home/user/Downloads/git-files/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90(1199):
> error #6623: The procedure name of the INTERFACE block conflicts with a
> name in the encompassing scoping unit.   [PCASMCREATESUBDOMAINS2D]
>
>   subroutine PCASMCreateSubdomains2D(a,b,c,d,e,f,g,h,i,z)
>
> -^
>
> I'm on the latest version of origin/main, but can't figure out how to fix
> that issue by myself. Therefore, I'd appreciate additional insight.
>

You have old build files in the tree. We changed the Fortran stubs to be
generated in the PETSC_ARCH tree so that you can build the stubs for
different branches in the same PETSc tree. You have old stubs in the src
tree. You can get rid of these using

  git clean -f -d -x

unless you have your own files in the source tree, in which case you need
to remove the ftn-auto-interfaces directories yourself.

  Thanks,

 Matt


> Thanks!
>
> Regards,
>
> Roland Richter
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] FEM Implementation of NS with SUPG Stabilization

[Matthew Knepley]

On Tue, Oct 10, 2023 at 9:34 PM Brandon Denton via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Good Evening,
>
> I am looking to implement a form of Navier-Stokes with SUPG Stabilization
> and shock capturing using PETSc's FEM infrastructure. In this
> implementation, I need access to the cell's shape function gradients and
> natural coordinate gradients for calculations within the point-wise
> residual calculations. How do I get these quantities at the quadrature
> points? The signatures for fo and f1 don't seem to contain this information.
>

Are you sure you need those? Darsh and I implemented SUPG without that. You
would need local second derivative information, which you can get using
-dm_ds_jet_degree 2. If you check in an example, I can go over it.

  Thanks,

 Matt


> Thank you in advance for your time.
> Brandon
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Parallel DMPlex

[Matthew Knepley]

On Tue, Oct 10, 2023 at 7:01 PM erdemguer  wrote:

>
> Hi,
> Sorry for my late response. I tried with your suggestions and I think I
> made a progress. But I still got issues. Let me explain my latest mesh
> routine:
>
>
>1. DMPlexCreateBoxMesh
>2. DMSetFromOptions
>3. PetscSectionCreate
>4. PetscSectionSetNumFields
>5. PetscSectionSetFieldDof
>6. PetscSectionSetDof
>7. PetscSectionSetUp
>8. DMSetLocalSection
>9. DMSetAdjacency
>10. DMPlexDistribute
>
>
> It's still not working but it's promising, if I call DMPlexGetDepthStratum
> for cells, I can see that after distribution processors have more cells.
>

Please send the output of DMPlexView() for each incarnation of the mesh.
What I do is put

  DMViewFromOptions(dm, NULL, "-dm1_view")

with a different string after each call.


> But I couldn't figure out how to decide where the ghost/processor boundary
> cells start.
>

Please send the actual code because the above is not specific enough. For
example, you will not have
"ghost cells" unless you partition with overlap. This is because by default
cells are the partitioned quantity,
so each process gets a unique set.

  Thanks,

  Matt


> In older mails I saw there is a function DMPlexGetHybridBounds but I
> think that function is deprecated. I tried to use,
> DMPlexGetCellTypeStratum as in ts/tutorials/ex11_sa.c but I'm getting -1
> as cEndInterior before and after distribution. I tried it for
> DM_POLYTOPE_FV_GHOST, DM_POLYTOPE_INTERIOR_GHOST polytope types. I also
> tried calling DMPlexComputeCellTypes before DMPlexGetCellTypeStratum but
> nothing changed. I think I can calculate the ghost cell indices using
> cStart/cEnd before & after distribution but I think there is a better way
> I'm currently missing.
>
> Thanks again,
> Guer.
>
> --- Original Message ---
> On Thursday, September 28th, 2023 at 10:42 PM, Matthew Knepley <
> knep...@gmail.com> wrote:
>
> On Thu, Sep 28, 2023 at 3:38 PM erdemguer via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
>> Hi,
>>
>> I am currently using DMPlex in my code. It runs serially at the moment,
>> but I'm interested in adding parallel options. Here is my workflow:
>>
>> Create a DMPlex mesh from GMSH.
>> Reorder it with DMPlexPermute.
>> Create necessary pre-processing arrays related to the mesh/problem.
>> Create field(s) with multi-dofs.
>> Create residual vectors.
>> Define a function to calculate the residual for each cell and, use SNES.
>> As you can see, I'm not using FV or FE structures (most examples do).
>> Now, I'm trying to implement this in parallel using a similar approach.
>> However, I'm struggling to understand how to create corresponding vectors
>> and how to obtain index sets for each processor. Is there a tutorial or
>> paper that covers this topic?
>>
>
> The intention was that there is enough information in the manual to do
> this.
>
> Using PetscFE/PetscFV is not required. However, I strongly encourage you
> to use PetscSection. Without this, it would be incredibly hard to do what
> you want. Once the DM has a Section, it can do things like automatically
> create vectors and matrices for you. It can redistribute them, subset them,
> etc. The Section describes how dofs are assigned to pieces of the mesh
> (mesh points). This is in the manual, and there are a few examples that do
> it by hand.
>
> So I suggest changing your code to use PetscSection, and then letting us
> know if things still do not work.
>
> Thanks,
>
> Matt
>
>> Thank you.
>> Guer.
>>
>> Sent with Proton Mail <https://proton.me/> secure email.
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Galerkin projection using petsc4py

[Matthew Knepley]

On Tue, Oct 10, 2023 at 5:34 PM Thanasis Boutsikakis <
thanasis.boutsika...@corintis.com> wrote:

> Hi all,
>
> Revisiting my code and the proposed solution from Pierre, I realized this
> works only in sequential. The reason is that PETSc partitions those
> matrices only row-wise, which leads to an error due to the mismatch between
> number of columns of A (non-partitioned) and the number of rows of Phi
> (partitioned).
>

Are you positive about this? P^T A P is designed to run in this scenario,
so either we have a bug or the diagnosis is wrong.

  Thanks,

 Matt


> """Experimenting with PETSc mat-mat multiplication"""
>
> import time
>
> import numpy as np
> from colorama import Fore
> from firedrake import COMM_SELF, COMM_WORLD
> from firedrake.petsc import PETSc
> from mpi4py import MPI
> from numpy.testing import assert_array_almost_equal
>
> from utilities import Print
>
> nproc = COMM_WORLD.size
> rank = COMM_WORLD.rank
>
> def create_petsc_matrix(input_array, sparse=True):
> """Create a PETSc matrix from an input_array
>
> Args:
> input_array (np array): Input array
> partition_like (PETSc mat, optional): Petsc matrix. Defaults to None.
> sparse (bool, optional): Toggle for sparese or dense. Defaults to True.
>
> Returns:
> PETSc mat: PETSc mpi matrix
> """
> # Check if input_array is 1D and reshape if necessary
> assert len(input_array.shape) == 2, "Input array should be 2-dimensional"
> global_rows, global_cols = input_array.shape
> size = ((None, global_rows), (global_cols, global_cols))
>
> # Create a sparse or dense matrix based on the 'sparse' argument
> if sparse:
> matrix = PETSc.Mat().createAIJ(size=size, comm=COMM_WORLD)
> else:
> matrix = PETSc.Mat().createDense(size=size, comm=COMM_WORLD)
> matrix.setUp()
>
> local_rows_start, local_rows_end = matrix.getOwnershipRange()
>
> for counter, i in enumerate(range(local_rows_start, local_rows_end)):
> # Calculate the correct row in the array for the current process
> row_in_array = counter + local_rows_start
> matrix.setValues(
> i, range(global_cols), input_array[row_in_array, :], addv=False
> )
>
> # Assembly the matrix to compute the final structure
> matrix.assemblyBegin()
> matrix.assemblyEnd()
>
> return matrix
>
> # 
> # EXP: Galerkin projection of an mpi PETSc matrix A with an mpi PETSc
> matrix Phi
> # A' = Phi.T * A * Phi
> # [k x k] <- [k x m] x [m x m] x [m x k]
> # 
>
> m, k = 100, 7
> # Generate the random numpy matrices
> np.random.seed(0) # sets the seed to 0
> A_np = np.random.randint(low=0, high=6, size=(m, m))
> Phi_np = np.random.randint(low=0, high=6, size=(m, k))
>
> # 
> # TEST: Galerking projection of numpy matrices A_np and Phi_np
> # 
> Aprime_np = Phi_np.T @ A_np @ Phi_np
> Print(f"MATRIX Aprime_np [{Aprime_np.shape[0]}x{Aprime_np.shape[1]}]")
> Print(f"{Aprime_np}")
>
> # Create A as an mpi matrix distributed on each process
> A = create_petsc_matrix(A_np, sparse=False)
>
> # Create Phi as an mpi matrix distributed on each process
> Phi = create_petsc_matrix(Phi_np, sparse=False)
>
> # Create an empty PETSc matrix object to store the result of the PtAP
> operation.
> # This will hold the result A' = Phi.T * A * Phi after the computation.
> A_prime = create_petsc_matrix(np.zeros((k, k)), sparse=False)
>
> # Perform the PtAP (Phi Transpose times A times Phi) operation.
> # In mathematical terms, this operation is A' = Phi.T * A * Phi.
> # A_prime will store the result of the operation.
> A_prime = A.ptap(Phi)
>
> Here is the error
>
> MATRIX mpiaij A [100x100]
> Assembled
>
> Partitioning for A:
>   Rank 0: Rows [0, 34)
>   Rank 1: Rows [34, 67)
>   Rank 2: Rows [67, 100)
>
> MATRIX mpiaij Phi [100x7]
> Assembled
>
> Partitioning for Phi:
>   Rank 0: Rows [0, 34)
>   Rank 1: Rows [34, 67)
>   Rank 2: Rows [67, 100)
>
> Traceback (most recent call last):
>   File "/Users/boutsitron/work/galerkin_projection.py", line 87, in
> 
> A_prime = A.ptap(Phi)
>   ^^^
>   File "petsc4py/PETSc/Mat.pyx", line 1525, in petsc4py.PETSc.Mat.ptap
> petsc4py.PETSc.Error: error code 60
> [0] MatPtAP() at
> /Users/boutsitron/firedrake/src/petsc/src/mat/interface/matrix.c:9896
> [0] MatProductSetFromOptions() at
> /Users/boutsitron/firedrake/src/petsc/src/mat/interface/matproduct.c:541
> [0] MatProductSetFromOptions_Private() at
> /Users/boutsitron/firedrake/src/petsc/src/mat/interface/matproduct.c:435
> [0] MatProductSetFromOptions_MPIAIJ() at
> /Users/boutsitron/firedrake/src/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:2372
> [0] MatProductSetFromOptions_MPIAIJ_PtAP() at
> /Users/boutsitron/firedrake/src/petsc/src/mat/impls/aij/mpi/mpimatmatmult.c:2266
> [0] Nonconforming object sizes
> [0] Matrix local dimensions are incompatible, Acol (0, 100) != Prow (0,34)
> Abort(1) on node 0 (rank 0 in comm 496): application called
> MPI_Abort(PYOP2

Re: [petsc-users] Scalability problem using PETSc with local installed OpenMPI

[Matthew Knepley]

On Tue, Oct 10, 2023 at 9:28 AM Gong Yujie 
wrote:

> Dear PETSc developers,
>
> I installed OpenMPI3 first and then installed PETSc with that mpi.
> Currently, I'm facing a scalability issue, in detail, I tested that using
> OpenMPI to calculate an addition of two distributed arrays and I get a good
> scalability. The problem is when I calculate the addition of two vectors in
> PETSc, I don't have any scalability. For the same size of the problem,
> PETSc costs a lot much time than merely using OpenMPI.
>
> My PETSc version is 3.16.0 and the version of OpenMPI is 3.1.4. Hope you
> can give me some suggestions.
>

1. For any performance question, we really need to see the output of
-log_view for each run.

2. I am not sure I understand your question. Vector addition does not
involve communication. Thus it will scale perfectly in the absence of load
imbalance.

  Thanks,

  Matt


> Best Regards,
> Yujie
>
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Compute integral using DMPlexComputeIntegralFEM

[Matthew Knepley]

On Fri, Sep 8, 2023 at 6:26 PM David Andrs  wrote:

> Hi all!
>
> I am trying to use DMPlexComputeIntegralFEM to compute an integral
> $\int_\Omega u d\Omega$. My domain is a square (-1, 1)^2 (2x2 QUAD4
> elements), I add first order Lagrange FE field on it, set the solution
> vector (computed by a previous simulation).
>
> The value I am seeing computed by PETSc is -4, but the hand-calculated
> value of this integral is -4.6. I also checked this in paraview using the
> ‘Integrate Variables’ filter and it also returns -4.6 (this was to double
> check that my hand-calculated value is correct).
>

Sorry it took so long. You caught me at a bad time.

Something must be wrong with your analytic integrals. Here is me doing them
by hand. You have a 3x3 vertex arrangement with coefficients

   1  0 -3
  -2 -1 -2
  -3  0  1

>From the symmetry, the integrals of the cells along each diagonal must be
equal. Now, the shape functions for Q_1 are

 (1 - x) yx y

  (1 - x)(1 - y) x (1 - y)

Thus the integral for the lower left cell is

  \int^1_0 dx \int^1_0 dy -3 + 3 x + y - 2 xy = -3 + 3/2 + 1/2 - 2/4 = -1.5

which is also the upper right cell. The integral for the lower right cell is

  \int^1_0 dx \int^1_0 dy x - y - 2 xy = 1/2 - 1/2 - 2/4 = -1/2

which is also the upper left cell. Thus we get -1.5 - 1.5 - 0.5 - 0.5 = -4,
which is what Plex gets.

  THanks,

Matt

So, I must be missing something obvious in my code. Attached is the minimal
> PETSc code to show what I am doing. This is against PETSc 3.19.4.
>
> Thanks in advance for your help,
>
> David
>
> --
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] How to do a precondition in SNES flow

[Matthew Knepley]

On Tue, Oct 3, 2023 at 3:05 PM Gong Ding  wrote:

> On 2023/10/4 02:47, Matthew Knepley wrote:
>
> On Tue, Oct 3, 2023 at 1:51 PM Gong Ding  wrote:
>
>> Hi all,
>>
>> I'd like to do a  special jacobian precondition during the snes
>> iteration, for which jacobian matrix and RHS vector must be modified
>> explicitly.
>>
>> In the SNESComputeJacobian, the preconditioner P is built after assembly
>> of jacobian matrix.
>>
>> I need to multiply P to J and RHS vector  explicitly as left precondition
>> before the solve stage of J*dx = rhs.
>>
>> What you are proposing is exactly what PETSc does with left
> preconditioning, multiplies both sides by the preconditioner. What do you
> want to change?
>
> I'd like to multiply precondition matrix into jacobian matrix, and do LU
> factorization to jacobian matrix. not with iterative method. Something like
>
> Kelley, C. T. "Newton's Method in Three Precisions." *arXiv preprint
> arXiv:2307.16051* (2023).
>
> BTW: does petsc have the plan to support multi-precision?
>
1. Tim is just solving the Newton equation with LU. You can do this using

  -pc_type lu

2. We do not support this kind of multi-precision. We had a plan to do
this, but no one to work on it. It does not seem to be a priority of users
so far.

  Thanks,

 Matt


>   Thanks,
>
>  Matt
>
>> However, I find that petsc evaluates function before jacobian, so P*RHS 
>> vector can not be processed at SNESComputeFunction.
>>
>> As a result,  I must find a hook function after SNESComputeJacobian and 
>> before the solve stage.
>>
>> Any suggest?
>>
>> Gong Ding
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] How to do a precondition in SNES flow

[Matthew Knepley]

On Tue, Oct 3, 2023 at 1:51 PM Gong Ding  wrote:

> Hi all,
>
> I'd like to do a  special jacobian precondition during the snes iteration,
> for which jacobian matrix and RHS vector must be modified explicitly.
>
> In the SNESComputeJacobian, the preconditioner P is built after assembly
> of jacobian matrix.
>
> I need to multiply P to J and RHS vector  explicitly as left precondition
> before the solve stage of J*dx = rhs.
>
> What you are proposing is exactly what PETSc does with left
preconditioning, multiplies both sides by the preconditioner. What do you
want to change?

  Thanks,

 Matt

> However, I find that petsc evaluates function before jacobian, so P*RHS 
> vector can not be processed at SNESComputeFunction.
>
> As a result,  I must find a hook function after SNESComputeJacobian and 
> before the solve stage.
>
> Any suggest?
>
> Gong Ding
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Parallel DMPlex

[Matthew Knepley]

On Thu, Sep 28, 2023 at 3:38 PM erdemguer via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi,
>
> I am currently using DMPlex in my code. It runs serially at the moment,
> but I'm interested in adding parallel options. Here is my workflow:
>
> Create a DMPlex mesh from GMSH.
> Reorder it with DMPlexPermute.
> Create necessary pre-processing arrays related to the mesh/problem.
> Create field(s) with multi-dofs.
> Create residual vectors.
> Define a function to calculate the residual for each cell and, use SNES.
> As you can see, I'm not using FV or FE structures (most examples do). Now,
> I'm trying to implement this in parallel using a similar approach. However,
> I'm struggling to understand how to create corresponding vectors and how to
> obtain index sets for each processor. Is there a tutorial or paper that
> covers this topic?
>

The intention was that there is enough information in the manual to do this.

Using PetscFE/PetscFV is not required. However, I strongly encourage you to
use PetscSection. Without this, it would be incredibly hard to do what you
want. Once the DM has a Section, it can do things like automatically create
vectors and matrices for you. It can redistribute them, subset them, etc.
The Section describes how dofs are assigned to pieces of the mesh (mesh
points). This is in the manual, and there are a few examples that do it by
hand.

So I suggest changing your code to use PetscSection, and then letting us
know if things still do not work.

  Thanks,

 Matt


> Thank you.
> Guer.
>
> Sent with Proton Mail  secure email.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] VecScale failed because the local size mismatch

[Matthew Knepley]

On Sun, Sep 24, 2023 at 3:49 PM Qiyue Lu  wrote:

> Hello,
> I have a vector (petsc_temp2) with size 1182 on my laptop without MPI.
> However, when I tried to call
> VecScale(petsc_temp2,  1.0/2.0);
> It failed with
> Cannot change/reset vector sizes to -1 local 1182 global after previously
> setting them to 1182 local 1182 global
>

1) Use PetscCall() to check all PETSc calls

2) Send the entire error output

  Thanks,

 Matt


> I wonder how this comes out and how the local size could be -1?
>
> Thanks
> Qiyue Lu
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Coupled system of equations on unstructured mesh

[Matthew Knepley]

On Sat, Sep 23, 2023 at 9:32 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> Sorry, you didn’t answer my first equation.
>
> 1)The M and N, in MatCreateBIJ, refer to the actual number of *global 
> *rows/cols
> or *global blocke*d rows/cols?
>

I did answer. It is global rows/cols.


> 2) The row indexes used in MatZeroRowsColumnsIS, would refer to the
> actual row or the blocked row?
>

Actual rows and cols.


> Below is your response to my previous email:
>
> “You would use the actual row numbers instead of block rows. You can use
> either.” How can I use either (or switch)?
>

I believe I meant you can use either the array or IS forms.

  Thanks,

     Matt


> Thanks,
>
> Karthik,
>
>
>
> *From: *Matthew Knepley 
> *Date: *Saturday, 23 September 2023 at 14:23
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sat, Sep 23, 2023 at 6:00 AM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> I just would like to clarify a couple of things.
>
>
>
> 1)The M and N, in MatCreateBIJ, refer to the actual number of global
> rows/cols or global blocked rows/cols?
>
>
>
> But d_nz and o_nz only refer to blocked rows.
>
>
>
> Yes.
>
>
>
> 2) In MatZeroRowsColumnsIS, you mentioned I could use the actual row
> numbers instead of block rows. Is there any option to SWITCH between the
> two, when using MatZeroRowsColumnsIS?
>
>
>
> I do not understand the question. When would switching happen?
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Thank you,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 22:03
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 5:00 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> I get it.
>
>
>
> Q1) In other words when using MatSetValuesBlocked, for bs = 2, I could
> pass all 64 entrees of the element matrix row-oriented by setting m = n  =
> 4 (along with the respective idxm and idxn)?
>
>
>
> Yes.
>
>
>
> Q2) However, at a given node variable u can fixed while v is free. In that
> case, while using MatZeroRowsColumns I need to know the global index of the
> variable u (i.e. row/column index in the system matrix A). Here, block size
> does not come in handy as it did when using MatSetValuesBlocked.
>
>
>
> You would use the actual row numbers instead of block rows. You can use
> either.
>
>
>
> Q3) When assembling the system matrix A in this way, would it allow me to
> use PCFIELDSPLIT for coupled systems?
>
>
>
> Yes.
>
>
>
>   Thanks,
>
>
>
>   Matt
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 21:23
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 4:16 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> Thank you. I almost got it and have a couple of questions.
>
>
>
> Let me summarize, I have an element Jacobian matrix of size 8 x 8 for
> QAUD4 (for variables u and v) ordered such that the degrees of freedom at a
> node are grouped together i.e. [u1 v1 u2 v2 u3 v3 u4 v4].
>
>
>
> Q1) But this ordering of element Jacobian matrix of size 8 x 8, does not
> have an on-diagonal and off-diagonal block structure? Does it not matter?
>
>
>
> It does have block structure in that each entry in your original matrix is
> now a 2x2 block entry in the matrix for 2 fields.
>
>
>
> When assembling the element Jacobian into the system matrix A, I would
> make use of MatSetValues. For each element, I *presume* m = 8 and n = 8
> along with idxm and idxn, are passed to MatSetValues. Meaning I have to
> figure out the global index (say idxm) set of size 8 corresponding to the
> system matrix A.
>
>
>
> What you can do is call
>
>
>
>   https://petsc.org/main/manualpages/Mat/MatSetValuesBlocked/
>
>
>
> and give the same indices that you would have given for the original
> 

Re: [petsc-users] Coupled system of equations on unstructured mesh

[Matthew Knepley]

On Sat, Sep 23, 2023 at 6:00 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> I just would like to clarify a couple of things.
>
>
>
> 1)The M and N, in MatCreateBIJ, refer to the actual number of global
> rows/cols or global blocked rows/cols?
>
>
>
> But d_nz and o_nz only refer to blocked rows.
>

Yes.


> 2) In MatZeroRowsColumnsIS, you mentioned I could use the actual row
> numbers instead of block rows. Is there any option to SWITCH between the
> two, when using MatZeroRowsColumnsIS?
>

I do not understand the question. When would switching happen?

  Thanks,

     Matt


> Thank you,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 22:03
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 5:00 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> I get it.
>
>
>
> Q1) In other words when using MatSetValuesBlocked, for bs = 2, I could
> pass all 64 entrees of the element matrix row-oriented by setting m = n  =
> 4 (along with the respective idxm and idxn)?
>
>
>
> Yes.
>
>
>
> Q2) However, at a given node variable u can fixed while v is free. In that
> case, while using MatZeroRowsColumns I need to know the global index of the
> variable u (i.e. row/column index in the system matrix A). Here, block size
> does not come in handy as it did when using MatSetValuesBlocked.
>
>
>
> You would use the actual row numbers instead of block rows. You can use
> either.
>
>
>
> Q3) When assembling the system matrix A in this way, would it allow me to
> use PCFIELDSPLIT for coupled systems?
>
>
>
> Yes.
>
>
>
>   Thanks,
>
>
>
>   Matt
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 21:23
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 4:16 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> Thank you. I almost got it and have a couple of questions.
>
>
>
> Let me summarize, I have an element Jacobian matrix of size 8 x 8 for
> QAUD4 (for variables u and v) ordered such that the degrees of freedom at a
> node are grouped together i.e. [u1 v1 u2 v2 u3 v3 u4 v4].
>
>
>
> Q1) But this ordering of element Jacobian matrix of size 8 x 8, does not
> have an on-diagonal and off-diagonal block structure? Does it not matter?
>
>
>
> It does have block structure in that each entry in your original matrix is
> now a 2x2 block entry in the matrix for 2 fields.
>
>
>
> When assembling the element Jacobian into the system matrix A, I would
> make use of MatSetValues. For each element, I *presume* m = 8 and n = 8
> along with idxm and idxn, are passed to MatSetValues. Meaning I have to
> figure out the global index (say idxm) set of size 8 corresponding to the
> system matrix A.
>
>
>
> What you can do is call
>
>
>
>   https://petsc.org/main/manualpages/Mat/MatSetValuesBlocked/
>
>
>
> and give the same indices that you would have given for the original
> matrix, but instead of a value at each location, we now have a 2x2 block.
>
>
>
> Q2) What does setting the block size bs = 2 accomplish? Because the way to
> assemble the Jacobian into the system matrix A seems to me no different
> than I did for the single variable scalar problem.
>
>
>
> This is answered above.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 20:18
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 3:15 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> I don’t see an option to give block size.
>
> https://petsc.org/release/manualpages/Mat/MatCreateAIJ/
>
>
>
> https://petsc.org/release/manualpages/Mat/MatSetBlockSize/
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
>
>
> Many thanks,

Re: [petsc-users] Problem with BLASdot in 3.19.4

[Matthew Knepley]

On Tue, Sep 19, 2023 at 7:04 AM Thuc Bui  wrote:

> Hi Barry,
>
>
>
> Visual Studio 2022 is the problem! The code linked to Petsc 3.18.6 built
> with VS 2022 also crashes at the same place. The same errors are shown
> below. I don’t remember for sure, but I don’t think I was able to configure
> Petsc 3.19.4 with VS 2019. However, I will still try that next.
>

It is so easy to hate VS. This suggests that VS is secretly linking to
another BLAS, perhaps with different calling semantics.

  Thanks,

 Matt


> Thanks for your help,
>
> Thuc
>
>
>
> [0]PETSC ERROR:
> 
>
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
>
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and
> https://petsc.org/release/faq/
>
> [0]PETSC ERROR: -  Stack Frames
> 
>
> [0]PETSC ERROR: The line numbers in the error traceback are not always
> exact.
>
> [0]PETSC ERROR: #1 BLASdot()
>
> [0]PETSC ERROR: #2 VecNorm_Seq() at
> D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\vec\vec\impls\seq\bvec2.c:216
>
> [0]PETSC ERROR: #3 VecNorm() at
> D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\vec\vec\interface\rvector.c:237
>
> [0]PETSC ERROR: #4 VecNormalize() at
> D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\vec\vec\interface\rvector.c:318
>
> [0]PETSC ERROR: #5 KSPGMRESCycle() at
> D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\impls\gmres\gmres.c:111
>
> [0]PETSC ERROR: #6 KSPSolve_GMRES() at
> D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\impls\gmres\gmres.c:228
>
> [0]PETSC ERROR: #7 KSPSolve_Private() at
> D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\interface\itfunc.c:899
>
> [0]PETSC ERROR: #8 KSPSolve() at
> D:\Users\bbwannabe\Documents\Petsc\petsc-3.18.6\src\ksp\ksp\interface\itfunc.c:1071
>
>
>
> job aborted:
>
> [ranks] message
>
>
>
> [0] application aborted
>
> aborting MPI_COMM_WORLD (comm=0x4400), error 59, comm rank 0
>
>
>
>
>
>
>
> *From:* petsc-users [mailto:petsc-users-boun...@mcs.anl.gov] *On Behalf
> Of *Thuc Bui
> *Sent:* Monday, September 18, 2023 4:24 PM
> *To:* 'Barry Smith'
> *Cc:* 'PETSc users list'
> *Subject:* Re: [petsc-users] Problem with BLASdot in 3.19.4
>
>
>
> Thanks a lot Barry, for getting back to me. Will do what you have
> suggested, and get back with the results.
>
>
>
> Best regards,
>
> Thuc
>
>
>
> *From:* Barry Smith [mailto:bsm...@petsc.dev]
> *Sent:* Monday, September 18, 2023 3:43 PM
> *To:* Thuc Bui
> *Cc:* PETSc users list
> *Subject:* Re: [petsc-users] Problem with BLASdot in 3.19.4
>
>
>
>
>
>   Ok, two things are being changed at the same time: the version  of PETSc
> and the version of Visual Studio.
>
>
>
>   Could you please try with the new Visual Studio version but the same
> older PETSc version? If that works could you try with the old Visual Studio
> version but the new PETSc version?
>
>
>
>   Barry
>
>
>
>
>
>
>
> On Sep 18, 2023, at 6:26 PM, Thuc Bui  wrote:
>
>
>
> Dear Petsc users and experts,
>
>
>
> If someone can direct me how to track this bug, I would really appreciate
> it.
>
>
>
> The Petsc DLL library version 3.19.4 was built on Windows 10 with Visual
> Studio 2022, and with Microsoft MPI 10.1.2 and Intel MKL 2020.3.279. The
> same code works fine with Petsc 3.18.6 using the same versions of MS MPI
> and Intel MKL, and built with Visual Studio 2019.
>
>
>
> When my code calls PetscCall(KSPSolve(...)), it gets to
> PetscCall(KSPGMRESCycle(...)), PetscCall(VecNormalize(VEC_VV(0), &res)),
> PetscCall(VecNorm(x, NORM_2, &norm)) and then crashes in VecNorm_Seq() at:
>
>
>
> PetscCallBLAS("BLASdot", ztmp[type == NORM_1_AND_2] =
> PetscSqrtReal(PetscRealPart(BLASdot_(&bn, xx, &one, xx, &one;
>
>
>
> I tried to step into BLASdot_, but was unable to. I assume BLASdot_ belong
> to Intel MKL, and its library does not contain debug information. After the
> code exits, the errors are shown below the hash line.
>
>
>
> Should I install the latest Intel MKL to work with Petsc 3.19.4?
>
>
>
> Many thanks in advance for your help,
>
> Thuc Bui
>
> Senior R&D Engineer
>
> Calabazas Creek Research, Inc
>
> (650) 948-5361
>
>
>
>
>
> ###
>
>
>
> [0]PETSC ERROR:
> 
>
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
>
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and
> https://petsc.org/release/faq/
>
> [0]PETSC ERROR: -  Stack Frames
> 
>
> [0]PETSC ERROR: The line numbers in the error traceback are not a

Re: [petsc-users] Looking for PETSc Example: Poisson Equation in 2D using FEM - scale upto 1000 procs

[Matthew Knepley]

On Mon, Sep 18, 2023 at 4:16 PM Khurana, Parv 
wrote:

> Hello PETSc users,
>
>
>
> Thank you for this very active community of users and the mailing list.
>
>
>
> I am looking for a PETSc example which solves the Poisson equation on a 2D
> domain using FEM (or HO-FEM if possible). I would like the following:
>
>1. The example should be formulated fully on PETSc, and to be solved
>with KSP objects in PETSc.
>2. The problem should scale up to a few hundred processors (ideally
>1000 procs).
>3. Ideally on a unit square with either Square or Triangle element
>discretization.
>4. There should be an option to specify Dirichlet/Neumann-type BCs on
>the boundaries.
>
>
>
> I was wondering if someone could point me to such an example as I am
> relatively new to PETSc – and I am trying to avoid reinventing the wheel. I
> have had a look at Examples 29,32,50 and 66 in the PETSc tutorials – while
> they are very close to what I need I am not sure if they scale to a few
> hundred processors. Furthermore, I am aware that I can formulate such a
> problem with FENICS/Firedrake with relative ease and these software
> interfaces with PETSc quite well. However, I am just trying to see if such
> an application already exists purely in PETSc.
>
>
>
It looks like Mark has everything working. However, if you have problems
getting that to run, I am here at ICL this week.

  Thanks,

 Matt


> Thanks, and Best
>
> Parv
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] [petsc-maint] compiler preprocessor for PETSc

[Matthew Knepley]

On Sat, Sep 16, 2023 at 10:20 AM Michael Wick 
wrote:

> Hi PETSc team:
>
> What is an appropriate compiler preprocessor to tell the code PETSc has
> been initialized?
>

https://petsc.org/main/manualpages/Sys/PetscInitialized/

  Thanks,

 Matt


> Mike
>
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Coupled system of equations on unstructured mesh

[Matthew Knepley]

On Sun, Sep 10, 2023 at 4:16 PM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> Thank you. I almost got it and have a couple of questions.
>
>
>
> Let me summarize, I have an element Jacobian matrix of size 8 x 8 for
> QAUD4 (for variables u and v) ordered such that the degrees of freedom at
> a node are grouped together i.e. [u1 v1 u2 v2 u3 v3 u4 v4].
>
>
>
> Q1) But this ordering of element Jacobian matrix of size 8 x 8, does not
> have an on-diagonal and off-diagonal block structure? Does it not matter?
>

It does have block structure in that each entry in your original matrix is
now a 2x2 block entry in the matrix for 2 fields.

When assembling the element Jacobian into the system matrix A, I would make
> use of MatSetValues. For each element, I *presume* m = 8 and n = 8 along
> with idxm and idxn, are passed to MatSetValues. Meaning I have to figure
> out the global index (say idxm) set of size 8 corresponding to the system
> matrix A.
>

What you can do is call

  https://petsc.org/main/manualpages/Mat/MatSetValuesBlocked/

and give the same indices that you would have given for the original
matrix, but instead of a value at each location, we now have a 2x2 block.


> Q2) What does setting the block size bs = 2 accomplish? Because the way to
> assemble the Jacobian into the system matrix A seems to me no different
> than I did for the single variable scalar problem.
>

This is answered above.

  Thanks,

 Matt


> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 20:18
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 3:15 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> I don’t see an option to give block size.
>
> https://petsc.org/release/manualpages/Mat/MatCreateAIJ/
>
>
>
> https://petsc.org/release/manualpages/Mat/MatSetBlockSize/
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
>
>
> Many thanks,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 20:09
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 3:01 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> I will solve the coupled problem in a non-linear fashion using SNES, so I
> will have one residual for each variable R(u1) and R(u2) and the resulting
> Jacobian matrix
>
>
>
> J =  [J11 J12
>
>  J21 J22] for each element.
>
>
>
> For first-order Lagrange elements (in 2D) will have four nodes, then J
> would be of size 8 x 8. Let's say the variables are u and v. The order
> would be [u1 u2 u3 u4 v1 v2 v3 v4].
>
>
>
> Or should I rearrange the above entrées in the element Jacobian so as to
> have [u1 v1 u2 v2 u3 v3 u4 v4] ordering? In that case, the Jacobian will
> not have block ordering I guess.
>
>
>
> This is the correct order to get block matrices.
>
>
>
> In my case, the discretization of the u and v are from the same element
> space.
>
> Can you please elaborate when you say, “Just enlarge the block size of
> your matrix”?
>
> I have never used block size before and not sure what it accomplishes.
>
> Do you mean to say, I should use MATBAIJ instead of MATAIJ?
>
>
>
> You do not need to change the type, just give the AIJ matrix a blocksize
> of 2.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 19:03
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 1:48 PM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
>
>
> I have so far solved scalar field problems using finite elements on a
> given (*unstructured*) mesh. I made use of MATMPIAIJ to create matrixes,
> MatCreateVecs(A, &b, &x) to create vectors, and MatZeroRowsColumnsIS to
> set boundary conditions.
>
>
>
> Now, I would like to solve a coupled system of equations for the
> quantities u1 an

Re: [petsc-users] Coupled system of equations on unstructured mesh

[Matthew Knepley]

On Sun, Sep 10, 2023 at 3:15 PM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> I don’t see an option to give block size.
>
> https://petsc.org/release/manualpages/Mat/MatCreateAIJ/
>

https://petsc.org/release/manualpages/Mat/MatSetBlockSize/

  Thanks,

 Matt


>
>
> Many thanks,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 20:09
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 3:01 PM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> I will solve the coupled problem in a non-linear fashion using SNES, so I
> will have one residual for each variable R(u1) and R(u2) and the resulting
> Jacobian matrix
>
>
>
> J =  [J11 J12
>
>  J21 J22] for each element.
>
>
>
> For first-order Lagrange elements (in 2D) will have four nodes, then J
> would be of size 8 x 8. Let's say the variables are u and v. The order
> would be [u1 u2 u3 u4 v1 v2 v3 v4].
>
>
>
> Or should I rearrange the above entrées in the element Jacobian so as to
> have [u1 v1 u2 v2 u3 v3 u4 v4] ordering? In that case, the Jacobian will
> not have block ordering I guess.
>
>
>
> This is the correct order to get block matrices.
>
>
>
> In my case, the discretization of the u and v are from the same element
> space.
>
> Can you please elaborate when you say, “Just enlarge the block size of
> your matrix”?
>
> I have never used block size before and not sure what it accomplishes.
>
> Do you mean to say, I should use MATBAIJ instead of MATAIJ?
>
>
>
> You do not need to change the type, just give the AIJ matrix a blocksize
> of 2.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 19:03
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 1:48 PM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
>
>
> I have so far solved scalar field problems using finite elements on a
> given (*unstructured*) mesh. I made use of MATMPIAIJ to create matrixes,
> MatCreateVecs(A, &b, &x) to create vectors, and MatZeroRowsColumnsIS to
> set boundary conditions.
>
>
>
> Now, I would like to solve a coupled system of equations for the
> quantities u1 and u2 on the (*unstructured*)  mesh. I.e., the matrix
> should get the double number of rows and columns,
>
>
>
> A = [A00  A01
>
> A10  A11]
>
>
>
> This is usually not a good way to think of it. This division means that
> all variables of one field come before
>
> all those of another. It is much more common to group together all the
> unknowns at a given point.
>
>
>
> You could, if the discretizations of u1 and u2 are the same, just enlarge
> the blocksize of your matrix. Then you set u1 and u2 for each vec location,
> or a 2x2 block for each Jacobian location.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> the vectors contain twice the number of entries (e.g. first all u1s and
> then all u2s). I would like to be sure that the entries of u1 and u2, which
> are associated with the same element are located on the same processor.
>
>
>
> Is a pre-defined structure already available within PETSc to enlarge such
> a single equation to store the entries of coupled equations?
>
>
>
> -\Delta u_1+c_{11} u_1+c_{12} u_2=f_1
>
> -\Delta u_2+c_{21} u_1+c_{22} u_2=f_2
>
>
>
> Would I still be able to use MatZeroRowsColumnsIS u1 and u2 independently
> to enforce boundary conditions? MatZeroRowsColumnsIS(A, is, 1, x, b);
>
>
>
> I don’t know where to begin. I have so far only been exposed to using 
> MATMPIAIJ
> and MatSetValues to create and assign values to matrix entries
> respectively. I would be grateful if you could provide the stepwise guide.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
> --
>
> *Karthik Chockalingam, Ph.D.*
>
> Senior Research Software Engineer
>
> High Performance Systems Engineering Group
>
> Hartree Centre | Science and Technology Facilities Council
>
> karthikeyan.chockalin...@stfc.ac.uk
>
>
>
>  *Error! Filena

Re: [petsc-users] Coupled system of equations on unstructured mesh

[Matthew Knepley]

On Sun, Sep 10, 2023 at 3:01 PM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> I will solve the coupled problem in a non-linear fashion using SNES, so I
> will have one residual for each variable R(u1) and R(u2) and the resulting
> Jacobian matrix
>
>
>
> J =  [J11 J12
>
>  J21 J22] for each element.
>
>
>
> For first-order Lagrange elements (in 2D) will have four nodes, then J
> would be of size 8 x 8. Let's say the variables are u and v. The order
> would be [u1 u2 u3 u4 v1 v2 v3 v4].
>
>
>
> Or should I rearrange the above entrées in the element Jacobian so as to
> have [u1 v1 u2 v2 u3 v3 u4 v4] ordering? In that case, the Jacobian will
> not have block ordering I guess.
>

This is the correct order to get block matrices.


> In my case, the discretization of the u and v are from the same element
> space.
>
> Can you please elaborate when you say, “Just enlarge the block size of
> your matrix”?
>
> I have never used block size before and not sure what it accomplishes.
>
> Do you mean to say, I should use MATBAIJ instead of MATAIJ?
>

You do not need to change the type, just give the AIJ matrix a blocksize of
2.

  Thanks,

 Matt


> Kind regards,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Sunday, 10 September 2023 at 19:03
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Coupled system of equations on unstructured
> mesh
>
> On Sun, Sep 10, 2023 at 1:48 PM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
>
>
> I have so far solved scalar field problems using finite elements on a
> given (*unstructured*) mesh. I made use of MATMPIAIJ to create matrixes,
> MatCreateVecs(A, &b, &x) to create vectors, and MatZeroRowsColumnsIS to
> set boundary conditions.
>
>
>
> Now, I would like to solve a coupled system of equations for the
> quantities u1 and u2 on the (*unstructured*)  mesh. I.e., the matrix
> should get the double number of rows and columns,
>
>
>
> A = [A00  A01
>
> A10  A11]
>
>
>
> This is usually not a good way to think of it. This division means that
> all variables of one field come before
>
> all those of another. It is much more common to group together all the
> unknowns at a given point.
>
>
>
> You could, if the discretizations of u1 and u2 are the same, just enlarge
> the blocksize of your matrix. Then you set u1 and u2 for each vec location,
> or a 2x2 block for each Jacobian location.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> the vectors contain twice the number of entries (e.g. first all u1s and
> then all u2s). I would like to be sure that the entries of u1 and u2, which
> are associated with the same element are located on the same processor.
>
>
>
> Is a pre-defined structure already available within PETSc to enlarge such
> a single equation to store the entries of coupled equations?
>
>
>
> -\Delta u_1+c_{11} u_1+c_{12} u_2=f_1
>
> -\Delta u_2+c_{21} u_1+c_{22} u_2=f_2
>
>
>
> Would I still be able to use MatZeroRowsColumnsIS u1 and u2 independently
> to enforce boundary conditions? MatZeroRowsColumnsIS(A, is, 1, x, b);
>
>
>
> I don’t know where to begin. I have so far only been exposed to using 
> MATMPIAIJ
> and MatSetValues to create and assign values to matrix entries
> respectively. I would be grateful if you could provide the stepwise guide.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
> --
>
> *Karthik Chockalingam, Ph.D.*
>
> Senior Research Software Engineer
>
> High Performance Systems Engineering Group
>
> Hartree Centre | Science and Technology Facilities Council
>
> karthikeyan.chockalin...@stfc.ac.uk
>
>
>
>  *Error! Filename not specified.*
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Coupled system of equations on unstructured mesh

[Matthew Knepley]

On Sun, Sep 10, 2023 at 1:48 PM Karthikeyan Chockalingam - STFC UKRI via
petsc-users  wrote:

> Hello,
>
>
>
> I have so far solved scalar field problems using finite elements on a
> given (*unstructured*) mesh. I made use of MATMPIAIJ to create matrixes,
> MatCreateVecs(A, &b, &x) to create vectors, and MatZeroRowsColumnsIS to
> set boundary conditions.
>
>
>
> Now, I would like to solve a coupled system of equations for the
> quantities u1 and u2 on the (*unstructured*)  mesh. I.e., the matrix
> should get the double number of rows and columns,
>
>
>
> A = [A00  A01
>
> A10  A11]
>

This is usually not a good way to think of it. This division means that all
variables of one field come before
all those of another. It is much more common to group together all the
unknowns at a given point.

You could, if the discretizations of u1 and u2 are the same, just enlarge
the blocksize of your matrix. Then you set u1 and u2 for each vec location,
or a 2x2 block for each Jacobian location.

  Thanks,

 Matt


> the vectors contain twice the number of entries (e.g. first all u1s and
> then all u2s). I would like to be sure that the entries of u1 and u2, which
> are associated with the same element are located on the same processor.
>
>
>
> Is a pre-defined structure already available within PETSc to enlarge such
> a single equation to store the entries of coupled equations?
>
>
>
> -\Delta u_1+c_{11} u_1+c_{12} u_2=f_1
>
> -\Delta u_2+c_{21} u_1+c_{22} u_2=f_2
>
>
>
> Would I still be able to use MatZeroRowsColumnsIS u1 and u2 independently
> to enforce boundary conditions? MatZeroRowsColumnsIS(A, is, 1, x, b);
>
>
>
> I don’t know where to begin. I have so far only been exposed to using 
> MATMPIAIJ
> and MatSetValues to create and assign values to matrix entries
> respectively. I would be grateful if you could provide the stepwise guide.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
> --
>
> *Karthik Chockalingam, Ph.D.*
>
> Senior Research Software Engineer
>
> High Performance Systems Engineering Group
>
> Hartree Centre | Science and Technology Facilities Council
>
> karthikeyan.chockalin...@stfc.ac.uk
>
>
>
>  [image: signature_3970890138]
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Slower performance using more MPI processes

[Matthew Knepley]

On Fri, Sep 8, 2023 at 4:53 PM Chris Hewson  wrote:

> Hi There,
>
> I am trying to solve a linear problem and am having an issue when I use
> more MPI processes with the KSPsolve slowing down considerably the more
> processes I add.
>
> The matrix itself is 620100 X 620100 with ~5 million non-zero entries, I
> am using petsc version 3.19.5 and have tried with a couple different
> versions of mpich getting the same behavior (v4.1.2 w/ device ch4:ofi and
> v3.3.2 w/ ch3:sock).
>
> In testing, I've noticed the following trend for speed for the KSPSolve
> function call:
> 1 core: 4042 ms
> 2 core: 7085 ms
> 4 core: 26573 ms
> 8 core: 65745 ms
> 16 core: 149283 ms
>
> This was all done on a single node machine w/ 16 non-hyperthreaded cores.
> We solve quite a few different matrices with PETSc using MPI and haven't
> noticed an impact like this on performance before.
>
> I am very confused by this and am a little stumped at the moment as to why
> this was happening. I've been using the KSPBCGS solver to solve the
> problem. I have tried with multiple different solvers and pre-conditioners
> (we usually don't use a pre-conditioner for this part of our code).
>
> It did seem that using the piped BCGS solver did help improve the parallel
> speed slightly (maybe 15%), but it still doesn't come close to the single
> threaded speed.
>
> I'll attach a link to a folder that contains the specific A, x and b
> matrices for this problem, as well as a main.cpp file that I was using for
> testing.
>
>
> https://drive.google.com/drive/folders/1CEDinKxu8ZbKpLtwmqKqP1ZIDG7JvDI1?usp=sharing
>
> I was testing this in our main code base, but don't include that here, and
> observe very similar speed results to the ones above. We do use Metis to
> graph partition in our own code and checked the vector and matrix
> partitioning and that all made sense. I could be doing the partitioning
> incorrectly in the example (not 100% sure how it works with the viewer/load
> functions).
>
> Any insight or thoughts on this would be greatly appreciated.
>

Send the output of -log_view for each case.

These are all memory bandwidth-limited operations. When you exhaust the
bandwidth you should
see the performance stagnate, not steeply decay as you do here. Something
definitely seems wrong.
The first step is sending us the logs.

  Thanks,

  Matt


> Thanks,
>
> *Chris Hewson*
> Senior Reservoir Simulation Engineer
> ResFrac
> +1.587.575.9792
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH

[Matthew Knepley]

On Thu, Sep 7, 2023 at 10:18 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> Yes, I had to zero both the Jacobian and Residual which fixed the SNES
> convergence problem!
>
>
>
> This is my first time solving using SNES. From the tutorial
>
> https://petsc4py.readthedocs.io/en/stable/manual/snes/
>
> I didn’t notice the Jacobian and Residual had to be zeroed.
>

Yes, these examples are all FD so the entries are not added to, but set
directly. The FEM examples do this zeroing out in the wrapper functions
like DMComputeResidualLocal, so it is hard to see.

  Thanks,

 Matt


> Thank you.
>
>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Thursday, 7 September 2023 at 14:32
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *Stefano Zampini , petsc-users@mcs.anl.gov
> 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> On Thu, Sep 7, 2023 at 9:28 AM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> Thank you for your response.
>
>
>
> I will test the Jacobian.
>
>
>
> The linear solve in the second step convergences. Is it possible that
> Jacobian had been zeroed out?
>
>
>
> I think you miss my point. Suppose you did not zero out the Jacobian when
> you started adding entries. Then you will get something close to twice the
> value. The linear system will solve fine, but the sep would be wrong.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Thursday, 7 September 2 ot023 at 14:19
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *Stefano Zampini , petsc-users@mcs.anl.gov
> 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> On Thu, Sep 7, 2023 at 8:49 AM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> I just noticed that the line search diverges even for the linear Poisson
> equation (where the Jacobian is a constant). I am not sure, what I am doing
> wrong.
>
>
>
> You need to test the Jacobian. We have facilities for this. Run using
>
>
>
>   --snes_test_jacobian
>
>
>
> This will tell you at each step whether the Jacobian matches the finite
> difference approximant. It looks like your Jacobian at the second step is
> wrong. Is it possible that you are not zeroing out the matrix before adding
> in updates? That would make the second step wrong.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Date: *Thursday, 7 September 2023 at 12:19
> *To: *Stefano Zampini 
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> I am trying to solve the non-linear scalar problem
>
> -\nabla \cdot \big( (1+u^2)\nabla u\big)=f
>
> Using finite element formulation.
>
>
>
> The linear iteration converged – does that mean my Jacobian is correct??
>
>
>
> I am hoping the non-linear residual goes down to 1-e11.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Stefano Zampini 
> *Date: *Thursday, 7 September 2023 at 11:49
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> The solver did not diverge.
>
> It was the line search that was not able to make further progress in
> minimizing the 2-norm of the residual.
>
> This is common in nonlinear solvers. It would help if you tell us what you
> are trying to solve.
>
> Note that at the first step, your residual norm is already 1.e-6. What
> kind of accuracy do you want?
>
>
>
>
>
> Il giorno gio 7 set 2023 alle ore 13:08 Karthikeyan Chockalingam - STFC
> UKRI via petsc-users  ha scritto:
>
> Hello,
>
>
>
> The non-linear solution diverged. The final solution seems right and I
> believe the Jacobian is correct (not 100% certain).
>
>
>
> I am not sure if I doing something wrong in the solver setting.
>
>
>
>
>
>  0 SNES Function norm 3.890991210938e-03
>
> 0 KSP Residual norm 9.037762538598e+00
>
> 1 KSP Residual norm 2.120375403775e-01
>
> 2 KSP Residual norm 5.155439334511e-03
>
> 3 KSP Residual norm 1.394364169369e-04
&g

Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH

[Matthew Knepley]

On Thu, Sep 7, 2023 at 9:28 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> Thank you for your response.
>
>
>
> I will test the Jacobian.
>
>
>
> The linear solve in the second step convergences. Is it possible that
> Jacobian had been zeroed out?
>

I think you miss my point. Suppose you did not zero out the Jacobian when
you started adding entries. Then you will get something close to twice the
value. The linear system will solve fine, but the sep would be wrong.

  Thanks,

 Matt


>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Thursday, 7 September 2 ot023 at 14:19
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *Stefano Zampini , petsc-users@mcs.anl.gov
> 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> On Thu, Sep 7, 2023 at 8:49 AM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> I just noticed that the line search diverges even for the linear Poisson
> equation (where the Jacobian is a constant). I am not sure, what I am doing
> wrong.
>
>
>
> You need to test the Jacobian. We have facilities for this. Run using
>
>
>
>   --snes_test_jacobian
>
>
>
> This will tell you at each step whether the Jacobian matches the finite
> difference approximant. It looks like your Jacobian at the second step is
> wrong. Is it possible that you are not zeroing out the matrix before adding
> in updates? That would make the second step wrong.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Date: *Thursday, 7 September 2023 at 12:19
> *To: *Stefano Zampini 
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> I am trying to solve the non-linear scalar problem
>
> -\nabla \cdot \big( (1+u^2)\nabla u\big)=f
>
> Using finite element formulation.
>
>
>
> The linear iteration converged – does that mean my Jacobian is correct??
>
>
>
> I am hoping the non-linear residual goes down to 1-e11.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Stefano Zampini 
> *Date: *Thursday, 7 September 2023 at 11:49
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> The solver did not diverge.
>
> It was the line search that was not able to make further progress in
> minimizing the 2-norm of the residual.
>
> This is common in nonlinear solvers. It would help if you tell us what you
> are trying to solve.
>
> Note that at the first step, your residual norm is already 1.e-6. What
> kind of accuracy do you want?
>
>
>
>
>
> Il giorno gio 7 set 2023 alle ore 13:08 Karthikeyan Chockalingam - STFC
> UKRI via petsc-users  ha scritto:
>
> Hello,
>
>
>
> The non-linear solution diverged. The final solution seems right and I
> believe the Jacobian is correct (not 100% certain).
>
>
>
> I am not sure if I doing something wrong in the solver setting.
>
>
>
>
>
>  0 SNES Function norm 3.890991210938e-03
>
> 0 KSP Residual norm 9.037762538598e+00
>
> 1 KSP Residual norm 2.120375403775e-01
>
> 2 KSP Residual norm 5.155439334511e-03
>
> 3 KSP Residual norm 1.394364169369e-04
>
> 4 KSP Residual norm 9.233543407204e-06
>
>   Linear solve converged due to CONVERGED_RTOL iterations 4
>
>
>
>   Line search: Using full step: fnorm 3.890991210938e-03 gnorm
> 7.701511565083e-06
>
>   1 SNES Function norm 7.701511565083e-06
>
> 0 KSP Residual norm 5.630229687829e-03
>
> 1 KSP Residual norm 1.030475601271e-04
>
> 2 KSP Residual norm 2.576454714319e-06
>
> 3 KSP Residual norm 6.669316846898e-08
>
> 4 KSP Residual norm 3.215810984829e-09
>
>   Linear solve converged due to CONVERGED_RTOL iterations 4
>
>
>
>   Line search: gnorm after quadratic fit 1.805500533481e-05
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 2.563759884284e-05 lambda=3.0804668685096816e-02
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 3.332721829751e-05 lambda=3.0804668685096817e-03
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 4.102754045833e-05 lambda=3.0804668685096822e-04
>
>   Line search: Cubi

Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH

[Matthew Knepley]

On Thu, Sep 7, 2023 at 8:49 AM Karthikeyan Chockalingam - STFC UKRI via
petsc-users  wrote:

> I just noticed that the line search diverges even for the linear Poisson
> equation (where the Jacobian is a constant). I am not sure, what I am doing
> wrong.
>

You need to test the Jacobian. We have facilities for this. Run using

  --snes_test_jacobian

This will tell you at each step whether the Jacobian matches the finite
difference approximant. It looks like your Jacobian at the second step is
wrong. Is it possible that you are not zeroing out the matrix before adding
in updates? That would make the second step wrong.

  Thanks,

 Matt


>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Date: *Thursday, 7 September 2023 at 12:19
> *To: *Stefano Zampini 
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> I am trying to solve the non-linear scalar problem
>
> -\nabla \cdot \big( (1+u^2)\nabla u\big)=f
>
> Using finite element formulation.
>
>
>
> The linear iteration converged – does that mean my Jacobian is correct??
>
>
>
> I am hoping the non-linear residual goes down to 1-e11.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Stefano Zampini 
> *Date: *Thursday, 7 September 2023 at 11:49
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] Non-linear solve: DIVERGED_LINE_SEARCH
>
> The solver did not diverge.
>
> It was the line search that was not able to make further progress in
> minimizing the 2-norm of the residual.
>
> This is common in nonlinear solvers. It would help if you tell us what you
> are trying to solve.
>
> Note that at the first step, your residual norm is already 1.e-6. What
> kind of accuracy do you want?
>
>
>
>
>
> Il giorno gio 7 set 2023 alle ore 13:08 Karthikeyan Chockalingam - STFC
> UKRI via petsc-users  ha scritto:
>
> Hello,
>
>
>
> The non-linear solution diverged. The final solution seems right and I
> believe the Jacobian is correct (not 100% certain).
>
>
>
> I am not sure if I doing something wrong in the solver setting.
>
>
>
>
>
>  0 SNES Function norm 3.890991210938e-03
>
> 0 KSP Residual norm 9.037762538598e+00
>
> 1 KSP Residual norm 2.120375403775e-01
>
> 2 KSP Residual norm 5.155439334511e-03
>
> 3 KSP Residual norm 1.394364169369e-04
>
> 4 KSP Residual norm 9.233543407204e-06
>
>   Linear solve converged due to CONVERGED_RTOL iterations 4
>
>
>
>   Line search: Using full step: fnorm 3.890991210938e-03 gnorm
> 7.701511565083e-06
>
>   1 SNES Function norm 7.701511565083e-06
>
> 0 KSP Residual norm 5.630229687829e-03
>
> 1 KSP Residual norm 1.030475601271e-04
>
> 2 KSP Residual norm 2.576454714319e-06
>
> 3 KSP Residual norm 6.669316846898e-08
>
> 4 KSP Residual norm 3.215810984829e-09
>
>   Linear solve converged due to CONVERGED_RTOL iterations 4
>
>
>
>   Line search: gnorm after quadratic fit 1.805500533481e-05
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 2.563759884284e-05 lambda=3.0804668685096816e-02
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 3.332721829751e-05 lambda=3.0804668685096817e-03
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 4.102754045833e-05 lambda=3.0804668685096822e-04
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 4.872893294880e-05 lambda=3.0804668685096822e-05
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 5.643043250787e-05 lambda=3.0804668685096822e-06
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 6.413194279696e-05 lambda=3.0804668685096827e-07
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 7.183345417492e-05 lambda=3.0804668685096828e-08
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 7.953496567312e-05 lambda=3.0804668685096829e-09
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 8.723647719173e-05 lambda=3.0804668685096831e-10
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 9.493798871875e-05 lambda=3.0804668685096832e-11
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 1.026395002516e-04 lambda=3.0804668685096835e-12
>
>   Line search: Cubic step no good, shrinking lambda, current gnorm
> 1.103410117889e-04 lambda=3.0804668685096835e-13
>
>   Line search: unable to find good step length! After 12 tries
>
>   Line search: fnorm=7.7015115650831560e-06,
> gnorm=1.1034101178892401e-04, ynorm=6.2052357872955976e-03,
> minlambda=9.9998e-13, lambda=3.0804668685096835e-13, initial
> slope=-5.9313318983096354e-11
>
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 1
>
>
>
>
>
> Thank you

Re: [petsc-users] Can we force SNES solver to do at least Newton step?

[Matthew Knepley]

On Sat, Sep 2, 2023 at 6:58 PM David Knezevic 
wrote:

> Hmm, it sounds like the convergence measure is bad. Maybe using a weighted
>> norm would be better?
>
>
> That's a good thought, I'd like to look into that idea too. Could you
> please give me some guidance on how to use a weighted norm in the
> convergence test? (Or are there any examples of doing that in the example
> suite?)
>

I guess the idea would be the following:

   You have exponential terms in your equation. You get a relatively small
residual, but unacceptable
   error. This leads me to believe that some part of the residual is
swamping other parts you care about.
   This should be confirmed by plotting the residual for one of these
0-iterate cases. To prevent this
   imbalance, you would scale the residual to give more weight to the
underrepresented parts, probably
   with an inverse of the exponential. This is just a more complicated form
of the diagonal scaling
   commonly used to give all fields the same scale.

  Thanks,

  Matt


> Thanks,
> David
>
> On Sat, Sep 2, 2023 at 5:54 PM Matthew Knepley  wrote:
>
>> On Sat, Sep 2, 2023 at 5:45 PM David Knezevic 
>> wrote:
>>
>>> OK, thanks, I'll look into the custom convergence test.
>>>
>>> I do not understand this comment. What do you mean by "inaccurate"?
>>>> Since we do not have the true solution, we usually say "inaccurate" for
>>>> large residual, but you already said that the residual is small. Why would
>>>> you want to do another iterate?
>>>
>>>
>>> I agree with your comments, but the specific case I'm considering is
>>> very numerically sensitive since it includes creep (which unfortunately has
>>> large exponential terms in it) which is the root cause of the issues I'm
>>> facing. Based on test cases with a known reference solution we're finding
>>> that we get inaccurate results due to steps with "zero iterations". We can
>>> fix this by tightening the tolerance but then we do an excessive number of
>>> iterations in other steps. So it seems to me that ensuring that we do at
>>> least one iteration will help here, so that's what I wanted to try.
>>>
>>
>> Hmm, it sounds like the convergence measure is bad. Maybe using a
>> weighted norm would be better?
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks again for your help.
>>>
>>> Best,
>>> David
>>>
>>> On Sat, Sep 2, 2023 at 3:23 PM Matthew Knepley 
>>> wrote:
>>>
>>>> On Sat, Sep 2, 2023 at 3:05 PM David Knezevic via petsc-users <
>>>> petsc-users@mcs.anl.gov> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm using the SNES solver for a plasticity model, and the issue I've
>>>>> run into is that in some time steps the solver terminates after "NL step 
>>>>> 0"
>>>>> since the initial residual (based on the solution from the previous time
>>>>> step) is below the specified tolerance.
>>>>>
>>>>> I gather that "NL step 0" only checks the residual and doesn't
>>>>> actually do a Newtown update, and hence it seems that this is leading to
>>>>> inaccurate results in some cases.
>>>>>
>>>>
>>>> I do not understand this comment. What do you mean by "inaccurate"?
>>>> Since we do not have the true solution, we usually say "inaccurate" for
>>>> large residual, but you already said that the residual is small.
>>>> Why would you want to do another iterate?
>>>>
>>>>
>>>>> I can of course specify a smaller convergence tolerance to avoid this
>>>>> issue, but I've found it difficult to find a smaller tolerance that works
>>>>> well in all cases (e.g. it leads to too many iterations or
>>>>> non-convergence). So instead what I would like to do is ensure that the
>>>>> solver does at least 1 Newton iteration instead of terminating at "NL step
>>>>> 0". Is there a way to enforce this behavior, e.g. by skipping "NL step 0",
>>>>> or specifying a "minimum number of iterations"? I didn't see anything like
>>>>> this in the documentation, so I was wondering if there are any suggestions
>>>>> on how to proceed for this.
>>>>>
>>>>
>>>> The easiest way to do this is to wri

Re: [petsc-users] Can we force SNES solver to do at least Newton step?

[Matthew Knepley]

On Sat, Sep 2, 2023 at 5:45 PM David Knezevic 
wrote:

> OK, thanks, I'll look into the custom convergence test.
>
> I do not understand this comment. What do you mean by "inaccurate"? Since
>> we do not have the true solution, we usually say "inaccurate" for large
>> residual, but you already said that the residual is small. Why would you
>> want to do another iterate?
>
>
> I agree with your comments, but the specific case I'm considering is very
> numerically sensitive since it includes creep (which unfortunately has
> large exponential terms in it) which is the root cause of the issues I'm
> facing. Based on test cases with a known reference solution we're finding
> that we get inaccurate results due to steps with "zero iterations". We can
> fix this by tightening the tolerance but then we do an excessive number of
> iterations in other steps. So it seems to me that ensuring that we do at
> least one iteration will help here, so that's what I wanted to try.
>

Hmm, it sounds like the convergence measure is bad. Maybe using a weighted
norm would be better?

  Thanks,

 Matt


> Thanks again for your help.
>
> Best,
> David
>
> On Sat, Sep 2, 2023 at 3:23 PM Matthew Knepley  wrote:
>
>> On Sat, Sep 2, 2023 at 3:05 PM David Knezevic via petsc-users <
>> petsc-users@mcs.anl.gov> wrote:
>>
>>> Hi all,
>>>
>>> I'm using the SNES solver for a plasticity model, and the issue I've run
>>> into is that in some time steps the solver terminates after "NL step 0"
>>> since the initial residual (based on the solution from the previous time
>>> step) is below the specified tolerance.
>>>
>>> I gather that "NL step 0" only checks the residual and doesn't actually
>>> do a Newtown update, and hence it seems that this is leading to inaccurate
>>> results in some cases.
>>>
>>
>> I do not understand this comment. What do you mean by "inaccurate"? Since
>> we do not have the true solution, we usually say "inaccurate" for large
>> residual, but you already said that the residual is small.
>> Why would you want to do another iterate?
>>
>>
>>> I can of course specify a smaller convergence tolerance to avoid this
>>> issue, but I've found it difficult to find a smaller tolerance that works
>>> well in all cases (e.g. it leads to too many iterations or
>>> non-convergence). So instead what I would like to do is ensure that the
>>> solver does at least 1 Newton iteration instead of terminating at "NL step
>>> 0". Is there a way to enforce this behavior, e.g. by skipping "NL step 0",
>>> or specifying a "minimum number of iterations"? I didn't see anything like
>>> this in the documentation, so I was wondering if there are any suggestions
>>> on how to proceed for this.
>>>
>>
>> The easiest way to do this is to write a custom convergence test that
>> looks like this
>>
>> PetscErrorCode SNESConvergedDefault(SNES snes, PetscInt it, PetscReal
>> xnorm, PetscReal snorm, PetscReal fnorm, SNESConvergedReason *reason, void
>> *dummy)
>> {
>>   PetscFunctionBeginUser;
>>   if (!it) {
>> *reason = SNES_CONVERGED_ITERATING;
>> PetscFunctionReturn(PETSC_SUCCESS);
>>   }
>>   PetscCall(SNESConvergedDefault(snes, it, xnorm, snorm, fnorm, reason,
>> dummy));
>>   PetscFunctionReturn(PETSC_SUCCESS);
>> }
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks,
>>> David
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Can we force SNES solver to do at least Newton step?

[Matthew Knepley]

On Sat, Sep 2, 2023 at 3:05 PM David Knezevic via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi all,
>
> I'm using the SNES solver for a plasticity model, and the issue I've run
> into is that in some time steps the solver terminates after "NL step 0"
> since the initial residual (based on the solution from the previous time
> step) is below the specified tolerance.
>
> I gather that "NL step 0" only checks the residual and doesn't actually do
> a Newtown update, and hence it seems that this is leading to inaccurate
> results in some cases.
>

I do not understand this comment. What do you mean by "inaccurate"? Since
we do not have the true solution, we usually say "inaccurate" for large
residual, but you already said that the residual is small.
Why would you want to do another iterate?


> I can of course specify a smaller convergence tolerance to avoid this
> issue, but I've found it difficult to find a smaller tolerance that works
> well in all cases (e.g. it leads to too many iterations or
> non-convergence). So instead what I would like to do is ensure that the
> solver does at least 1 Newton iteration instead of terminating at "NL step
> 0". Is there a way to enforce this behavior, e.g. by skipping "NL step 0",
> or specifying a "minimum number of iterations"? I didn't see anything like
> this in the documentation, so I was wondering if there are any suggestions
> on how to proceed for this.
>

The easiest way to do this is to write a custom convergence test that looks
like this

PetscErrorCode SNESConvergedDefault(SNES snes, PetscInt it, PetscReal
xnorm, PetscReal snorm, PetscReal fnorm, SNESConvergedReason *reason, void
*dummy)
{
  PetscFunctionBeginUser;
  if (!it) {
*reason = SNES_CONVERGED_ITERATING;
PetscFunctionReturn(PETSC_SUCCESS);
  }
  PetscCall(SNESConvergedDefault(snes, it, xnorm, snorm, fnorm, reason,
dummy));
  PetscFunctionReturn(PETSC_SUCCESS);
}

  Thanks,

 Matt


> Thanks,
> David
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Configuring PETSc on Mac M1

[Matthew Knepley]

You need to send us configure.log so that we can see what the failure
message is.

  Thanks,

Matt

On Fri, Sep 1, 2023 at 9:56 AM Giselle Sosa Jones 
wrote:

> Hello,
>
> I am trying to install PETSc on my M1 Mac and I keep encountering the
> following error when configuring:
>
> Unknown Fortran name mangling: Are you sure the C and Fortran compilers
> are compatible?
>
>   Perhaps one is 64 bit and one is 32 bit?
>
>
> Any ideas on how to fix this? All my compilers are 64 bit. I am using
> gfortran and gcc (installed with brew).
>
> Thank you in advance.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Error while building PETSc with MATLAB

[Matthew Knepley]

On Tue, Aug 29, 2023 at 9:08 AM Satish Balay via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Send configure.log, make.log from both petsc-3.13 and 3.17 [or 3.19].
>
> [you can gzip them to make the logs friendly to mailing list - or send
> them to petsc-maint]
>
> And does test suite work with 3.17? [or 3.19?]
>

David Wells is working on this. The change is that petscversion.h now
includes petscconf.h which means you need all the include flags, but
Libmesh does not get the flags right.

  Thanks,

Matt


> Satish
>
> On Tue, 29 Aug 2023, INTURU SRINIVAS 20PHD0548 via petsc-users wrote:
>
> > I am sharing the make.log file while building petsc-3.13.4 with Matlab.
> > Please find the attachment and do the needful.
> >
> > On Tue, Aug 29, 2023 at 10:19 AM INTURU SRINIVAS 20PHD0548 <
> > inturu.srinivas2...@vitstudent.ac.in> wrote:
> >
> > > I tried with petsc-3.17.5. During building of libmesh, the error shows
> > > petsc was not found
> > >
> > > On Mon, Aug 28, 2023 at 9:43 PM Satish Balay 
> wrote:
> > >
> > >> https://ibamr.github.io/linux says petsc-3.17
> > >>
> > >> Here you are using 3.13
> > >>
> > >> Can you retry with petsc-3.17.5?
> > >>
> > >> Satish
> > >>
> > >> On Mon, 28 Aug 2023, INTURU SRINIVAS 20PHD0548 via petsc-users wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > I want to build PETSc with MATLAB for working on the simulation
> using
> > >> IBAMR
> > >> > open software. While building the PETSc, using the following
> > >> >
> > >> > export PETSC_DIR=$PWD
> > >> > export PETSC_ARCH=linux-debug
> > >> > ./configure \
> > >> >   --CC=$HOME/sfw/linux/openmpi/4.1.4/bin/mpicc \
> > >> >   --CXX=$HOME/sfw/linux/openmpi/4.1.4/bin/mpicxx \
> > >> >   --FC=$HOME/sfw/linux/openmpi/4.1.4/bin/mpif90 \
> > >> >   --with-debugging=1 \
> > >> >   --download-hypre=1 \
> > >> >   --download-fblaslapack=1 \
> > >> >   --with-x=0 \
> > >> >   --with-matlab-dir=/usr/local/MATLAB/R2020b/
> > >> >   --with-matlab-engine=1
> > >> >   --with-matlab-engine-dir=/usr/local/MATLAB/R2020b/extern/engines/
> > >> >
> > >> > make -j4
> > >> > make -j4 test
> > >> >
> > >> > I got the following error
> > >> > CLINKER
> > >> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test
> > >> > /usr/bin/ld:
> > >> >
> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >> > function `EvaluateResidual':
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:32:
> > >> > undefined reference to `PetscMatlabEnginePut'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:33:
> > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:35:
> > >> > undefined reference to `PetscMatlabEngineGet'
> > >> > /usr/bin/ld:
> > >> >
> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >> > function `EvaluateJacobian':
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:46:
> > >> > undefined reference to `PetscMatlabEnginePut'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:47:
> > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:49:
> > >> > undefined reference to `PetscMatlabEngineGet'
> > >> > /usr/bin/ld:
> > >> >
> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >> > function `TaoPounders':
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:75:
> > >> > undefined reference to `PetscMatlabEngineGet'
> > >> > /usr/bin/ld:
> > >> >
> linux-debug/tests/tao/leastsquares/tutorials/matlab/matlab_ls_test.o: in
> > >> > function `main':
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:126:
> > >> > undefined reference to `PetscMatlabEngineCreate'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:127:
> > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:139:
> > >> > undefined reference to `PetscMatlabEngineEvaluate'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:140:
> > >> > undefined reference to `PetscMatlabEngineGetArray'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutorials/matlab/matlab_ls_test.c:142:
> > >> > undefined reference to `PetscMatlabEngineGetArray'
> > >> > /usr/bin/ld:
> > >> >
> > >>
> /home/vit/sfw/petsc/3.13.4/src/tao/leastsquares/tutor

Re: [petsc-users] unable to distribute rows of a matrix across processors while loading

[Matthew Knepley]

On Sun, Aug 27, 2023 at 8:30 AM maitri ksh  wrote:

> Hi,
> I am using MatSetSizes() followed by MatLoad() to distribute the rows of a
> *sparse* matrix (*48x48*) across the processors. But it seems
> like the entire matrix is getting loaded in each of the processors instead
> of distributing it. What am I missing here?
>
> *code snippet:*
> Mat Js;
> MatType type = MATMPIAIJ;
> PetscViewer viewerJ;
> PetscCall(PetscViewerBinaryOpen(PETSC_COMM_WORLD, "Js.dat",
> FILE_MODE_READ, &viewerJ));
> PetscCall(MatCreate(PETSC_COMM_WORLD, &Js));
> PetscCall(MatSetSizes(Js, PETSC_DECIDE, PETSC_DECIDE, N, N));
> PetscCall(MatSetType(Js, type));
> PetscCall(MatLoad(Js, viewerJ));
> PetscCall(PetscViewerDestroy(&viewerJ));
> PetscCall(MatGetLocalSize(Js, &m, &n));
> PetscCall(MatGetSize(Js, &M, &N));
> PetscPrintf(PETSC_COMM_WORLD, "Js,Local rows: %d, Local columns:
> %d\n", m, n);
>

If this was really PETSC_COMM_WORLD, then this print statement would only
output once, but you have 4 lines. Therefore, MPI is messed up. I am
guessing you used an 'mpirun' or 'mpiexec' that is from a different MPI,
and therefore the launch did not work correctly. If you had PETSc install
MPI, then the correct one is in

  ${PETSC_DIR}/${PETSC_ARCH}/bin/mpiexec

  Thanks,

 Matt


>
> *Output *of  'mpiexec -n 4 ./check':
> Js,Local rows: 48, Local columns: 48
> Js,Local rows: 48, Local columns: 48
> Js,Local rows: 48, Local columns: 48
> Js,Local rows: 48, Local columns: 48
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] C++11 related issue

[Matthew Knepley]

On Fri, Aug 25, 2023 at 7:05 AM maitri ksh  wrote:

> Thanks Matt for the clarification, I used gcc11.2 instead and the
> petsc-configuration worked out fine. But now, I am facing an issue while
> doing the 'make check' of the libraries. The error points to some
> unbalanced or mismatch of double-quote (")
>
> /usr/bin/bash: -c: line 6: unexpected EOF while looking for matching `"'
> /usr/bin/bash: -c: line 7: syntax error: unexpected end of file
>
> The error seems to be related to a shell script, I am unable to debug the
> error,  I looked for instances where 'bash -c' is being used, I couldn't
> locate any. How should I proceed?
>

You can do the check by hand:

  cd $PETSC_DIR
  cd src/snes/tutorials
  make ex5
  ./.ex5 -snes_monitor

  Thanks,

 Matt


> On Thu, Aug 24, 2023 at 1:37 PM Matthew Knepley  wrote:
>
>> On Thu, Aug 24, 2023 at 6:10 AM maitri ksh  wrote:
>>
>>> I was facing a problem while compiling a code (which earlier, got
>>> compiled successfully using the same petsc set up), the problem was related
>>> to compilers. I decided to reconfigure petsc but ran into errors which are
>>> related to non-compliance of the compiler with  'C++11'. I had faced this
>>> issue earlier when I was installing Petsc and I had it resolved by using a
>>> newer version of compiler (openmpi-4.1.5).
>>>
>>
>> OpenMPI is not a compiler. It is an implementation of MPI that produces
>> compiler wrappers. Your actual compiler appears to be GCC 4.8.2:
>>
>>   Output from compiling with  -std=c++11
>> In file included from /usr/include/c++/4.8.2/algorithm:62:0,
>>  from
>> /tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:9:
>> /usr/include/c++/4.8.2/bits/stl_algo.h: In instantiation of
>> ‘_RandomAccessIterator std::__unguarded_partition(_RandomAccessIterator,
>> _RandomAccessIterator, const _Tp&, _Compare) [with _RandomAccessIterator =
>> __gnu_cxx::__normal_iterator*,
>> std::vector > >; _Tp = std::unique_ptr;
>> _Compare = main()::__lambda0]’:
>> /usr/include/c++/4.8.2/bits/stl_algo.h:2296:78:   required from
>> ‘_RandomAccessIterator
>> std::__unguarded_partition_pivot(_RandomAccessIterator,
>> _RandomAccessIterator, _Compare) [with _RandomAccessIterator =
>> __gnu_cxx::__normal_iterator*,
>> std::vector > >; _Compare = main()::__lambda0]’
>> /usr/include/c++/4.8.2/bits/stl_algo.h:2337:62:   required from ‘void
>> std::__introsort_loop(_RandomAccessIterator, _RandomAccessIterator, _Size,
>> _Compare) [with _RandomAccessIterator =
>> __gnu_cxx::__normal_iterator*,
>> std::vector > >; _Size = long int; _Compare =
>> main()::__lambda0]’
>> /usr/include/c++/4.8.2/bits/stl_algo.h:5499:44:   required from ‘void
>> std::sort(_RAIter, _RAIter, _Compare) [with _RAIter =
>> __gnu_cxx::__normal_iterator*,
>> std::vector > >; _Compare = main()::__lambda0]’
>> /tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:58:119:   required
>> from here
>> /usr/include/c++/4.8.2/bits/stl_algo.h:2263:35: error: no match for call
>> to ‘(main()::__lambda0) (std::unique_ptr&, const
>> std::unique_ptr&)’
>> while (__comp(*__first, __pivot))
>>^
>> /tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:58:42: note:
>> candidates are:
>>  std::sort(vector.begin(), vector.end(), [](std::unique_ptr &a,
>> std::unique_ptr &b) { return *a < *b; });
>>   ^
>> In file included from /usr/include/c++/4.8.2/algorithm:62:0,
>>  from
>> /tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:9:
>> /usr/include/c++/4.8.2/bits/stl_algo.h:2263:35: note: bool
>> (*)(std::unique_ptr&, std::unique_ptr&) 
>> while (__comp(*__first, __pivot))
>>
>> This compiler was released almost 10 years ago and has incomplete support
>> for C++11.
>>
>>
>>> Now, I am trying to use the same compiler (openmpi-4.1.5) to reconfigure
>>> petsc but the old issue (related to 'C++11') pops up. I used a code that
>>> was available online
>>> <https://stackoverflow.com/questions/5047971/how-do-i-check-for-c11-support>
>>> to check if the present compiler supports C++11, and it shows it does
>>> support.
>>>
>>
>> You may not have read to the bottom of the answer, but it tells you how
>> to check for complete support for C++11 and this compiler definitely does
>> not have it.
>>
>>   Thanks,
>>
>>   

Re: [petsc-users] C++11 related issue

[Matthew Knepley]

On Thu, Aug 24, 2023 at 6:10 AM maitri ksh  wrote:

> I was facing a problem while compiling a code (which earlier, got compiled
> successfully using the same petsc set up), the problem was related to
> compilers. I decided to reconfigure petsc but ran into errors which are
> related to non-compliance of the compiler with  'C++11'. I had faced this
> issue earlier when I was installing Petsc and I had it resolved by using a
> newer version of compiler (openmpi-4.1.5).
>

OpenMPI is not a compiler. It is an implementation of MPI that produces
compiler wrappers. Your actual compiler appears to be GCC 4.8.2:

  Output from compiling with  -std=c++11
In file included from /usr/include/c++/4.8.2/algorithm:62:0,
 from /tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:9:
/usr/include/c++/4.8.2/bits/stl_algo.h: In instantiation of
‘_RandomAccessIterator std::__unguarded_partition(_RandomAccessIterator,
_RandomAccessIterator, const _Tp&, _Compare) [with _RandomAccessIterator =
__gnu_cxx::__normal_iterator*,
std::vector > >; _Tp = std::unique_ptr;
_Compare = main()::__lambda0]’:
/usr/include/c++/4.8.2/bits/stl_algo.h:2296:78:   required from
‘_RandomAccessIterator
std::__unguarded_partition_pivot(_RandomAccessIterator,
_RandomAccessIterator, _Compare) [with _RandomAccessIterator =
__gnu_cxx::__normal_iterator*,
std::vector > >; _Compare = main()::__lambda0]’
/usr/include/c++/4.8.2/bits/stl_algo.h:2337:62:   required from ‘void
std::__introsort_loop(_RandomAccessIterator, _RandomAccessIterator, _Size,
_Compare) [with _RandomAccessIterator =
__gnu_cxx::__normal_iterator*,
std::vector > >; _Size = long int; _Compare =
main()::__lambda0]’
/usr/include/c++/4.8.2/bits/stl_algo.h:5499:44:   required from ‘void
std::sort(_RAIter, _RAIter, _Compare) [with _RAIter =
__gnu_cxx::__normal_iterator*,
std::vector > >; _Compare = main()::__lambda0]’
/tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:58:119:   required from
here
/usr/include/c++/4.8.2/bits/stl_algo.h:2263:35: error: no match for call to
‘(main()::__lambda0) (std::unique_ptr&, const
std::unique_ptr&)’
while (__comp(*__first, __pivot))
   ^
/tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:58:42: note: candidates
are:
 std::sort(vector.begin(), vector.end(), [](std::unique_ptr &a,
std::unique_ptr &b) { return *a < *b; });
  ^
In file included from /usr/include/c++/4.8.2/algorithm:62:0,
 from /tmp/petsc-hl_0r720/config.setCompilers/conftest.cc:9:
/usr/include/c++/4.8.2/bits/stl_algo.h:2263:35: note: bool
(*)(std::unique_ptr&, std::unique_ptr&) 
while (__comp(*__first, __pivot))

This compiler was released almost 10 years ago and has incomplete support
for C++11.


> Now, I am trying to use the same compiler (openmpi-4.1.5) to reconfigure
> petsc but the old issue (related to 'C++11') pops up. I used a code that
> was available online
> 
> to check if the present compiler supports C++11, and it shows it does
> support.
>

You may not have read to the bottom of the answer, but it tells you how to
check for complete support for C++11 and this compiler definitely does not
have it.

  Thanks,

 Matt


> I have attached the '*configure.log*' herewith for your reference.
> Can anyone suggest how to resolve/work-around this issue?
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Runtime options to activate GPU offloading of PETSc solvers

[Matthew Knepley]

On Wed, Aug 23, 2023 at 9:27 AM TROTIGNON Laurent 
wrote:

> Hello all,
>
> In the on line docs of PETSc, I found this paragraph :
>
>
>
> « PETSc uses a single source programming model where solver back-ends are
> selected as *runtime* options and configuration options with no changes
> to the API.
>
> Users should (ideally) never have to change their source code to take
> advantage of new backend implementations. »
>
>
>
> I am looking for an example of runtime options that enable activation of
> GPU offloading of PETSc solvers.
>
> I am not sure that runtime options for GPU offloading are currently
> available ?
>
> I am currently using petsc/3.19.2 configured with cuda and nvhpc/22.2.
>
>
Here is the FAQ on this: https://petsc.org/main/faq/#doc-faq-gpuhowto

The short answer is that GPU offloading is enabled by changing the types of
vectors and matrices, so that all the operations are executed on the GPU.
The solvers just organize those operations, so they need no changes.

  Thanks,

 Matt


> Best regards
>
> Laurent
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU

[Matthew Knepley]

On Tue, Aug 22, 2023 at 2:54 PM Vanella, Marcos (Fed) via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi Junchao, both the slurm scontrol show job_id -dd and looking at
> CUDA_VISIBLE_DEVICES does not provide information about which MPI process
> is associated to which GPU in the node in our system. I can see this with
> nvidia-smi, but if you have any other suggestion using slurm I would like
> to hear it.
>
> I've been trying to compile the code+Petsc in summit, but have been having
> all sorts of issues related to spectrum-mpi, and the different compilers
> they provide (I tried gcc, nvhpc, pgi, xl. Some of them don't handle
> Fortran 2018, others give issues of repeated MPI definitions, etc.).
>

The PETSc configure examples are in the repository:


https://gitlab.com/petsc/petsc/-/blob/main/config/examples/arch-olcf-summit-opt.py?ref_type=heads

Thanks,

  Matt


> I also wanted to ask you, do you know if it is possible to compile PETSc
> with the xl/16.1.1-10 suite?
>
> Thanks!
>
> I configured the library --with-cuda and when compiling I get a
> compilation error with CUDAC:
>
> CUDAC arch-linux-opt-xl/obj/src/sys/classes/random/impls/curand/curand2.o
> In file included from
> /autofs/nccs-svm1_home1/vanellam/Software/petsc/src/sys/classes/random/impls/curand/
> curand2.cu:1:
> In file included from
> /autofs/nccs-svm1_home1/vanellam/Software/petsc/include/petsc/private/randomimpl.h:5:
> In file included from
> /autofs/nccs-svm1_home1/vanellam/Software/petsc/include/petsc/private/petscimpl.h:7:
> In file included from
> /autofs/nccs-svm1_home1/vanellam/Software/petsc/include/petscsys.h:44:
> In file included from
> /autofs/nccs-svm1_home1/vanellam/Software/petsc/include/petscsystypes.h:532:
> In file included from /sw/summit/cuda/11.7.1/include/thrust/complex.h:24:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/detail/config.h:23:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/detail/config/config.h:27:
> /sw/summit/cuda/11.7.1/include/thrust/detail/config/cpp_dialect.h:112:6:
> warning: Thrust requires at least Clang 7.0. Define
> THRUST_IGNORE_DEPRECATED_CPP_DIALECT to suppress this message.
> [-W#pragma-messages]
>  THRUST_COMPILER_DEPRECATION(Clang 7.0);
>  ^
> /sw/summit/cuda/11.7.1/include/thrust/detail/config/cpp_dialect.h:101:3:
> note: expanded from macro 'THRUST_COMPILER_DEPRECATION'
>   THRUST_COMP_DEPR_IMPL(Thrust requires at least REQ. Define
> THRUST_IGNORE_DEPRECATED_CPP_DIALECT to suppress this message.)
>   ^
> /sw/summit/cuda/11.7.1/include/thrust/detail/config/cpp_dialect.h:95:38:
> note: expanded from macro 'THRUST_COMP_DEPR_IMPL'
> #  define THRUST_COMP_DEPR_IMPL(msg) THRUST_COMP_DEPR_IMPL0(GCC warning
> #msg)
>  ^
> /sw/summit/cuda/11.7.1/include/thrust/detail/config/cpp_dialect.h:96:40:
> note: expanded from macro 'THRUST_COMP_DEPR_IMPL0'
> #  define THRUST_COMP_DEPR_IMPL0(expr) _Pragma(#expr)
>^
> :141:6: note: expanded from here
>  GCC warning "Thrust requires at least Clang 7.0. Define
> THRUST_IGNORE_DEPRECATED_CPP_DIALECT to suppress this message."
>  ^
> In file included from
> /autofs/nccs-svm1_home1/vanellam/Software/petsc/src/sys/classes/random/impls/curand/
> curand2.cu:2:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/transform.h:721:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/detail/transform.inl:27:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/system/detail/generic/transform.h:104:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/system/detail/generic/transform.inl:19:
> In file included from /sw/summit/cuda/11.7.1/include/thrust/for_each.h:277:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/detail/for_each.inl:27:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/system/detail/adl/for_each.h:42:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/system/cuda/detail/for_each.h:35:
> In file included from
> /sw/summit/cuda/11.7.1/include/thrust/system/cuda/detail/util.h:36:
> In file included from
> /sw/summit/cuda/11.7.1/include/cub/detail/device_synchronize.cuh:19:
> In file included from /sw/summit/cuda/11.7.1/include/cub/util_arch.cuh:36:
> /sw/summit/cuda/11.7.1/include/cub/util_cpp_dialect.cuh:123:6: warning:
> CUB requires at least Clang 7.0. Define CUB_IGNORE_DEPRECATED_CPP_DIALECT
> to suppress this message. [-W#pragma-messages]
>  CUB_COMPILER_DEPRECATION(Clang 7.0);
>  ^
> /sw/summit/cuda/11.7.1/include/cub/util_cpp_dialect.cuh:112:3: note:
> expanded from macro 'CUB_COMPILER_DEPRECATION'
>   CUB_COMP_DEPR_IMPL(CUB requires at least REQ. Define
> CUB_IGNORE_DEPRECATED_CPP_DIALECT to suppress this message.)
>   ^
> /sw/summit/cuda/11.7.1/include/cub/util_cpp_dialect.cuh:106:35: note:
> expanded from macro 'CUB_COMP_DEPR_IMPL'
> #  define CUB_COMP_DEPR_IMPL(msg) CUB_COMP_DEPR_IMPL0(GCC warning #msg)
>  

Re: [petsc-users] (Sub) KSP initial guess with PCREDISTRIBUTE

[Matthew Knepley]

On Mon, Aug 21, 2023 at 4:06 AM Jonas Lundgren via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Dear PETSc users,
>
>
>
> I have a problem regarding the setting of initial guess to KSP when using
> PCREDISTRIBUTE as the preconditioner. (The reason to why I use
> PCREDISTRIBUTE is because I have a lot of fixed DOF in my problem, and
> PCREDISTRIBUTE successfully reduces the problem size and therefore speeds
> up the solving).
>
>
>
> First, some details:
>
> -  I use a version of PETSc 3.19.1
>
> -  The KSP I use is KSPPREONLY, as suggested in the manual pages
> of PCREDISTRIBUTE:
> https://petsc.org/release/manualpages/PC/PCREDISTRIBUTE/
>
> -  I use KSPBCGSL as sub-KSP. I can perfectly well solve my
> problem using this as my main KSP, but the performance is much worse than
> when using it as my sub-KSP (under KSPPREONLY+PCREDISTRIBUTE) due to the
> amount of fixed DOF
>
> -  I am first solving a state problem, then an adjoint problem
> using the same linear operator.
>
> -  The adjoint vector is used as sensitivity information to
> update a design. After the design update, the state+adjoint problems are
> solved again with a slightly updated linear operator. This is done for
> hundreds of (design) iteration steps
>
> -  I want the initial guess for the state problem to be the state
> solution from the previous (design) iteration, and same for the adjoint
> problem
>
> -  I am aware of the default way of setting a custom initial
> guess: KSPSetInitialGuessNonzero(ksp, PETSC_TRUE) together with providing
> the actual guess in the x vector in the call to KSPSolve(ksp, b, x)
>
>
>
> The main problem is that PCREDISTRIBUTE internally doesn’t use the input
> solution vector (x) when calling KSPSolve() for the sub-KSP. It zeroes out
> the solution vector (x) when starting to build x = diag(A)^{-1} b in the
> beginning of PCApply_Redistribute(), and uses red->x as the solution
> vector/initial guess when calling KSPSolve(). Therefore, I cannot reach the
> sub-KSP with an initial guess.
>
>
>
> Additionally, KSPPREONLY prohibits the use of having a nonzero initial
> guess (the error message says “it doesn’t make sense”). I guess I can
> remove the line raising this error and recompile the PETSc libraries, but
> it still won’t solve the previously mentioned problem, which seems to be
> the hard nut to crack.
>
>
>
> So far, I have found out that if I create 2 KSP object, one each for the
> state and adjoint problems, it is enough with calling
> KSPSetInitialGuessNonzero(subksp, PETSC_TRUE) on the subksp. It seems as if
> the variable red->x in PCApply_Redistribute() is kept untouched in memory
> between calls to the main KSP and therefore is used as (non-zero) initial
> guess to the sub-KSP. This has been verified by introducing
> PetscCall(PetscObjectCompose((PetscObject)pc,"redx",(PetscObject)red->x));
> in PCApply_Redistribute(), recompiling the PETSc library, and then
> inserting a corresponding PetscObjectQuery((PetscObject)pc, "redx",
> (PetscObject *)&redx); in my own program source file.
>
>
>
> However, I would like to only create 1 KSP to be used with both the state
> and adjoint problems (same linear operator), for memory reasons.
>

I  would like to understand this better. Are you trying to save the
temporary vector memory for another solver? For BCGSL, this is only a few
vectors. It should be much less than the matrices.

  Thanks,

 Matt


> When I do this, the initial guesses are mixed up between the two problems:
> the initial guess for the adjoint problem is the solution to the state
> problem in the current design iteration, and the initial guess for the
> state problem is the solution to the adjoint problem in the previous design
> iteration. These are very bad guesses that increases the time to
> convergence in each state/adjoint solve.
>
>
>
> So, the core of the problem (as far as I can understand) is that I want to
> control the initial guess red->x in PCApply_Redistribute().
>
>
>
> The only solution I can think of is to include a call to
> PetscObjectQuery() in PCApply_Redistribute() to obtain a vector with the
> initial guess from my main program. And then I need to keep track of the
> initial guesses for my both problems in my main program myself (minor
> problem). This is maybe not the neatest way, and I do not know if this
> approach affects the performance negatively? Maybe one call each to
> PetscObjectQuery() and PetscObjectCompose() per call to
> PCApply_Redistribute() is negligible?
>
>
>
> Is there another (and maybe simpler) solution to this problem? Maybe I can
> SCATTER_FORWARD the input x vector in PCApply_Redistribute() before it is
> zeroed out, together with allowing for non-zero initial guess in KSPPREONLY?
>
>
>
> Any help is welcome!
>
>
>
>
>
> Best regards,
>
> Jonas Lundgren
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any re

Re: [petsc-users] MetSetValues not found

[Matthew Knepley]

On Sun, Aug 20, 2023 at 6:48 AM Sanjay Govindjee  wrote:

> I recently got a copy of E.Bueler's petsc book and was converting the
> programs to fortran for my students and have run into a compile error that
> I can not figure out.  I comment out the MatSetValues line the code
> compiles fine.
>
> When I run make on vecmatkspf.F90, I get the error:
>
>42 |   PetscCallA(MatSetValues(A,1,i,4,j,aA,INSERT_VALUES,ierr))
>   | 1
> Error: There is no specific subroutine for the generic 'matsetvalues' at
> (1)
>
> Your 'i' is a PetscInt, whereas it should be an array like 'j'.

  Thanks,

 Matt


> My petsc set works, runs with my FEA code and compiles and runs the fortran
> programs in the petsc tutorials directories just fine.  And in particular,
> compiles
> and runs the C versions from Bueler.
>
> I've attached the make file and the source.  I must be doing something
> obviously wrong but I can not spot it.
>
> -sanjay
>
> --
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Filling non-zero values of a Petsc matrix using numpy arrays with non-zero indices and values (petsc4py)

[Matthew Knepley]

On Sun, Aug 20, 2023 at 12:53 AM Erik Kneller via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi All,
>
> Thank you for the recommendations. The first option provided by Matthew
> works in petsc4py but required some additional computations to re-organize
> information. I developed an approach that was easier to build within my
> current system using the option provided by Hong along with some other
> posts I found on the user forum. See below for an example of how I
> integrated Numba and petcs4py so the filling of matrix elements on each
> processor is done using the optimized machine code produced by Numba (you
> can eliminate the cleaning step if the number of non-zero elements for each
> row is well defined). Scaling and overall performance is now satisfactory.
>
> I do have another question. In order to make this work I create the Petsc
> matrix 'A' twice, first to get local ownership and second to define the
> local elements for a particular processor:
>
> Algorithm
> 
> (1) Create a Petsc matrix 'A' and set size and type
> (2) Get the local row start and end for matrix 'A'
> (3) Define the local non-zero coefficients for the rows owned by processor
> using a Numba JIT-compiled loop and store result in a csr matrix defined
> using Scipy.
> (4) Redefine Petsc matix 'A' using the the local csr elements define in
> step 3.
> (5) Begin and end assembly
> (6) Define RHS and initial solution vectors
> (7) Solve system
>  (see code example below for details)
>
> Steps (1) and (4) appear redundant and potentially sub-optimal from a
> performance perspective (perhaps due to my limited experience with
> petscy4py). Is there a better approach in terms of elegance and
> performance?
>

In 1), if you just want the default ownership, you can call

  https://petsc.org/main/manualpages/Sys/PetscSplitOwnership/

which is what the Mat is calling underneath.

  Thanks,

Matt


> Also, if this is the optimal syntax could someone elaborate on what
> exactly is occurring behind the seen in terms of memory allocation?
>
> Thank you all again for your guidance and insights.
>
> Modified Version of Dalcin's 2D Poisson Example with Numba
>
> --
> import sys
> from typing import Tuple
> import numpy.typing as npt
> import numpy as np
> from numba import njit
> import petsc4py
> petsc4py.init(sys.argv)
> from petsc4py import PETSc
> import scipy.sparse as sps
>
>
> def main() -> None:
> xnodes = 8000
> nnz_max = 10
>
> ynodes = xnodes
> N = xnodes*ynodes
> dx = 1.0/(xnodes + 1)
>
> A = PETSc.Mat()
> A.create(comm=PETSc.COMM_WORLD)
> A.setSizes((xnodes*ynodes, xnodes*ynodes))
> A.setType(PETSc.Mat.Type.AIJ)
>
> rstart, rend = A.getOwnershipRange()
> Ascipy = build_csr_matrix(
> rstart, rend, nnz_max, dx, xnodes, ynodes)
> csr=(
> Ascipy.indptr[rstart:rend+1] - Ascipy.indptr[rstart],
> Ascipy.indices[Ascipy.indptr[rstart]:Ascipy.indptr[rend]],
> Ascipy.data[Ascipy.indptr[rstart]:Ascipy.indptr[rend]]
> )
> A = PETSc.Mat().createAIJ(size=(N,N), csr=csr)
>
> A.assemblyBegin()
> A.assemblyEnd()
>
> ksp = PETSc.KSP()
> ksp.create(comm=A.getComm())
> ksp.setType(PETSc.KSP.Type.CG)
> ksp.getPC().setType(PETSc.PC.Type.GAMG)
>
> ksp.setOperators(A)
> ksp.setFromOptions()
>
> x, b = A.createVecs()
> b.set(1.0)
>
> ksp.solve(b, x)
>
>
> def build_csr_matrix(
> rstart:int,
> rend:int,
> nnz_max:int,
> dx:float,
> xnodes:int,
> ynodes:int
> ):
> Anz, Arow, Acol = build_nonzero_arrays(
> rstart, rend, nnz_max, dx, xnodes, ynodes
> )
> N = xnodes*ynodes
> Ls = sps.csr_matrix((Anz, (Arow,Acol)), shape=(N,N), dtype=np.float64)
> return Ls
>
>
> def build_nonzero_arrays(
> rstart:int,
> rend:int,
> nnz_max:int,
> dx:float,
> xnodes:int,
> ynodes:int
> ) -> Tuple[
> npt.NDArray[np.float64],
> npt.NDArray[np.float64],
> npt.NDArray[np.float64]
> ]:
> nrows_local = (rend - rstart) + 1
> Anz_ini = np.zeros((nnz_max*nrows_local), dtype=np.float64)
> Arow_ini = np.zeros((nnz_max*nrows_local), dtype=np.int32)
> Acol_ini = np.zeros((nnz_max*nrows_local), dtype=np.int32)
> icount_nz = define_nonzero_values(
> rstart, rend, dx, xnodes, ynodes, Anz_ini, Arow_ini, Acol_ini
> )
> (
> Anz, Arow, Acol
> ) = clean_nonzero_arrays(icount_nz, Anz_ini, Arow_ini, Acol_ini)
> return Anz, Arow, Acol
>
>
> @njit
> def define_nonzero_values(
> rstart:int,
> rend:int,
> dx:float,
> xnodes:int,
> ynodes:int,
> Anz:npt.NDArray[np.float64],
> Arow:npt.NDArray[np.int64],
> Acol:npt.NDArray[np.int64]
> ) -> int:
> """ Fill matrix A
> """
> icount_nz = 0
> for row in rang

Re: [petsc-users] Filling non-zero values of a Petsc matrix using numpy arrays with non-zero indices and values (petsc4py)

[Matthew Knepley]

On Fri, Aug 18, 2023 at 12:49 AM Erik Kneller via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi All,
>
> I need to fill non-zero values of a Petsc matrix via petsc4py for the
> domain defined by A.getOwnershipRange() using three Numpy arrays: (1)
> array containing row indices of non-zero value, (2) array containing column
> indices of non-zero values and (3) array containing the non-zero matrix
> values. How can one perform this type of filling operation in petsc4py?
> The method A.setValues does not appear to allow this since it only works on
> an individual matrix element or a block of matrix elements.
>
> I am using Numpy arrays since they can be computed in loops optimized
> using Numba on each processor. I also cannot pass the Petsc matrix to a
> Numba compiled function since type information cannot be inferred. I
> absolutely need to avoid looping in standard Python to define Petsc matrix
> elements due to performance issues. I also need to use a standard petscy4py
> method and avoid writing new C or Fortran wrappers to minimize language
> complexity.
>
> Example Algorithm Building on Lisandro Dalcin's 2D Poisson Example:
>
> --
> comm = PETSc.COMM_WORLD
> rank = comm.getRank()
>
> dx = 1.0/(xnodes + 1) # xnodes is the number of nodes in the x and
> y-directions of the grid
> nnz_max = 5 # max number of non-zero values per row
>
> A = PETSc.Mat()
> A.create(comm=PETSc.COMM_WORLD)
> A.setSizes((xnodes*ynodes, xnodes*ynodes))
> A.setType(PETSc.Mat.Type.AIJ)
> A.setPreallocationNNZ(nnz_max)
>
> rstart, rend = A.getOwnershipRange()
>
> # Here Anz, Arow and Acol are vectors with size equal to the number of
> non-zero values
> Anz, Arow, Acol = build_nonzero_numpy_arrays_using_numba(rstart, rend,
> nnz_max, dx, xnodes, ynodes)
>
> A.setValues(Arow, Acol, Anz) # <--- This does not work.
>

I see at least two options

  https://petsc.org/main/manualpages/Mat/MatCreateSeqAIJWithArrays/

or

  https://petsc.org/main/manualpages/Mat/MatSetPreallocationCOOLocal/
  https://petsc.org/main/manualpages/Mat/MatSetValuesCOO/

  Thanks,

 Matt


> A.assemblyBegin()
> A.assemblyEnd()
> ksp = PETSc.KSP()
> ksp.create(comm=A.getComm())
> ksp.setType(PETSc.KSP.Type.CG)
> ksp.getPC().setType(PETSc.PC.Type.GAMG)
> ksp.setOperators(A)
> ksp.setFromOptions()
> x, b = A.createVecs()
> b.set(1.0)
>
> ksp.solve(b, x)
>
> Regards,
> Erik Kneller, Ph.D.
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DM/DS crash after update to 3.19 (Fortran)

[Matthew Knepley]

On Tue, Aug 15, 2023 at 2:06 AM Martin Diehl 
wrote:

> Dear PETSc team,
>
> my simulation crashes after updating from 3.18.5 to 3.19.4.
>
> The error message is attached, so is the main code. The mesh (variable
> named geomMesh) is read with DMPlexCreateFromFile in a different part
> of the code).
> I did not start serious debugging yet in the hope that you can point me
> into the right direction having recent changes in FE/DS in mind.
>
> If this does not ring a bell, I'll have a look with a PETSc debug
>

I put in code to force all fields to use the same quadrature by default
since many people had problems
with this. For some reason, it does not guess the quadrature for your mesh
correctly. You can turn this off
using

  -petscds_force_quad 0

  Thanks,

 Matt


> build.
>
> many thanks in advance,
> Martin
> --
> KU Leuven
> Department of Computer Science
> Department of Materials Engineering
> Celestijnenlaan 200a
> 3001 Leuven, Belgium
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] error related to 'valgrind' when using MatView

[Matthew Knepley]

On Thu, Aug 10, 2023 at 2:30 AM maitri ksh  wrote:

> I am unable to understand what possibly went wrong with my code, I could
> load a matrix (large sparse matrix) into petsc, write it out and read it
> back into Matlab but when I tried to use MatView to see the matrix-info, it
> produces error of some 'corrupt argument, #valgrind'. Can anyone please
> help?
>

You use

  viewer = PETSC_VIEWER_STDOUT_WORLD

but then you Destroy() that viewer. You should not since you did not create
it.

  THanks,

 Matt


> Maitri
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] DMPlex bug in src/snes/tutorials/ex17.c

[Matthew Knepley]

On Wed, Aug 9, 2023 at 11:09 AM Ilya Fursov 
wrote:

> Hello,
>
> I have a problem running src/snes/tutorials/ex17.c in parallel,
> given the specific runtime options (these options are actually taken from
> the test example ex17_3d_q3_trig_elas).
>
> *The serial version works fine:*
> ./ex17 -dm_plex_box_faces 1,1,1 -sol_type elas_trig -dm_plex_dim 3
> -dm_plex_simplex 0 -displacement_petscspace_degree 3 -dm_refine 0
> -convest_num_refine 1 -snes_convergence_estimate -snes_monitor
>
> *The parallel version fails:*
> mpirun -n 2 ./ex17 -dm_plex_box_faces 1,1,1 -sol_type elas_trig
> -dm_plex_dim 3 -dm_plex_simplex 0 -displacement_petscspace_degree 3
> -dm_refine 0 -convest_num_refine 1 -snes_convergence_estimate -snes_monitor
> *with the error message (--with-debugging=1):*
> [0]PETSC ERROR: - Error Message
> --
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Cell 0 has not been assigned a cell type
>

Yes, I need to put some more checks for corner cases. You have a 1 element
mesh on 2 processes, so there is no cell 0, but I am expecting at least one
cell during setup. Give

  -dm_refine 1

or

  -dm_plex_box_faces 2,2,2

or something like that.

  Thanks,

 Matt


> [0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could
> be the program crashed before they were used or a spelling mistake, etc!
> [0]PETSC ERROR:   Option left: name:-convest_num_refine value: 1 source:
> command line
> [0]PETSC ERROR:   Option left: name:-displacement_petscspace_degree value:
> 3 source: command line
> [0]PETSC ERROR:   Option left: name:-snes_convergence_estimate (no value)
> source: command line
> [0]PETSC ERROR:   Option left: name:-snes_monitor (no value) source:
> command line
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.19.4, unknown
> [0]PETSC ERROR: ./ex17 on a DEBUG named ilya-HP-Notebook by ilya Wed Aug
>  9 19:23:02 2023
> [0]PETSC ERROR: Configure options PETSC_ARCH=DEBUG
> PETSC_DIR=/home/ilya/build/petsc-09aug-debug/petsc
> --with-blaslapack-dir=/home/ilya/progs/OpenBLAS-0.3.21
> --with-mpi-dir=/home/ilya/progs/openmpi-4.1.4 --with-debugging=1
> --download-hdf5 --download-hypre --download-chaco --download-metis
> --download-parmetis --download-suitesparse --download-moab --download-mumps
> --download-scalapack --download-superlu --download-superlu_dist
> --download-triangle --download-ml --download-giflib --download-libjpeg
> --download-libpng --download-zlib --download-spai --download-tchem
> --download-party --download-cmake --download-hwloc --download-ptscotch
> --download-revolve --download-cams --download-spai
> [0]PETSC ERROR: #1 DMPlexGetCellType() at
> /home/ilya/build/petsc-09aug-debug/petsc/src/dm/impls/plex/plex.c:5169
> [0]PETSC ERROR: #2 SetupFE() at ex17.c:621
> [0]PETSC ERROR: #3 main() at ex17.c:654
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -convest_num_refine 1 (source: command line)
> [0]PETSC ERROR: -displacement_petscspace_degree 3 (source: command line)
> [0]PETSC ERROR: -dm_plex_box_faces 1,1,1 (source: command line)
> [0]PETSC ERROR: -dm_plex_dim 3 (source: command line)
> [0]PETSC ERROR: -dm_plex_simplex 0 (source: command line)
> [0]PETSC ERROR: -dm_refine 0 (source: command line)
> [0]PETSC ERROR: -snes_convergence_estimate (source: command line)
> [0]PETSC ERROR: -snes_monitor (source: command line)
> [0]PETSC ERROR: -sol_type elas_trig (source: command line)
> [0]PETSC ERROR: End of Error Message ---send entire
> error message to petsc-ma...@mcs.anl.gov--
>
> *And when --with-debugging=0, it fails with segmentation violation:*
> [0]PETSC ERROR:
> 
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and
> https://petsc.org/release/faq/
> [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and
> run
> [0]PETSC ERROR: to get more information on the crash.
> [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is
> causing the crash.
>
> Regards,
> Ilya
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Something went wrong with PETSc after installing ParaView

[Matthew Knepley]

On Sun, Aug 6, 2023 at 3:03 AM K. Wu  wrote:

> Dear Matthew,
>
> Thanks for your kind help, please see in attachment the configure.log file
> for PETSc.
>

Okay, you told PETSc to build MPICH, so you should use

  /lhome/kai/Documents/petsc/linux-c-debug-directsolver/bin/mpiexec -n
 ./myprog

It shows this is the report at the end of the log.

  Thanks,

 Matt

Regards,
> Kai
>
> Matthew Knepley  于2023年8月5日周六 16:19写道：
>
>> On Sat, Aug 5, 2023 at 10:10 AM K. Wu  wrote:
>>
>>> Dear Matthew,
>>>
>>> Thanks for your reply!
>>>
>>> Is there any way that I can choose to use the previous MPI installation
>>> used to build PETSc?
>>>
>>
>> First we would need to know what that was. You can send configure.log,
>> which will have that information in it.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Regards,
>>> Kai
>>>
>>> Matthew Knepley  于2023年8月5日周六 14:23写道：
>>>
>>>> On Sat, Aug 5, 2023 at 3:22 AM K. Wu  wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Good day!
>>>>>
>>>>> After installing ParaView on my desktop, PETSc starts to work
>>>>> anomalously even after reconfiguration:
>>>>> 1. If I use mpirun (frequently used before), it seems that now all the
>>>>> processors will run the program independently without communication. While
>>>>> mpiexec seems to work properly.
>>>>>
>>>>
>>>> Yes, this indicates that the "mpirun" is from a different installation
>>>> of MPI than "mpiexec", which seems to be from the MPI installation used to
>>>> build PETSc. Probably the Paraview package installed its own MPI package.
>>>>
>>>>
>>>>>
>>>>> 2. The Makefile (as attached) which works fine before starts to
>>>>> complain:
>>>>> make: *** No rule to make target 'chkopts', needed by 'test'.  Stop.
>>>>>
>>>>
>>>> The 'chkopts' target is defined in the toplevel PETSc makefile, but it
>>>> is deprecated. You can take it out.
>>>>
>>>>   Thanks,
>>>>
>>>>  Matt
>>>>
>>>>
>>>>> Thanks for your kind help!
>>>>>
>>>>> Best regards,
>>>>> Kai
>>>>>
>>>>>
>>>>> PETSC_DIR=~/petsc
>>>>> PETSC_ARCH=arch-linux-c-debug
>>>>> CFLAGS = -I.
>>>>> FFLAGS=
>>>>> CPPFLAGS=-I.
>>>>> FPPFLAGS=
>>>>> LOCDIR=
>>>>> EXAMPLESC=
>>>>> EXAMPLESF=
>>>>> MANSEC=
>>>>> CLEANFILES=
>>>>> NP=
>>>>>
>>>>>
>>>>> include ${PETSC_DIR}/lib/petsc/conf/variables
>>>>> include ${PETSC_DIR}/lib/petsc/conf/rules
>>>>> include ${PETSC_DIR}/lib/petsc/conf/test
>>>>>
>>>>> test: ex2-2.o chkopts
>>>>> rm -rf topopt
>>>>> -${CLINKER} -o test ex2-2.o ${PETSC_SYS_LIB}
>>>>> ${RM} ex2-2.o
>>>>> rm -rf *.o
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Something went wrong with PETSc after installing ParaView

[Matthew Knepley]

On Sat, Aug 5, 2023 at 10:10 AM K. Wu  wrote:

> Dear Matthew,
>
> Thanks for your reply!
>
> Is there any way that I can choose to use the previous MPI installation
> used to build PETSc?
>

First we would need to know what that was. You can send configure.log,
which will have that information in it.

  Thanks,

 Matt


> Regards,
> Kai
>
> Matthew Knepley  于2023年8月5日周六 14:23写道：
>
>> On Sat, Aug 5, 2023 at 3:22 AM K. Wu  wrote:
>>
>>> Hi all,
>>>
>>> Good day!
>>>
>>> After installing ParaView on my desktop, PETSc starts to work
>>> anomalously even after reconfiguration:
>>> 1. If I use mpirun (frequently used before), it seems that now all the
>>> processors will run the program independently without communication. While
>>> mpiexec seems to work properly.
>>>
>>
>> Yes, this indicates that the "mpirun" is from a different installation of
>> MPI than "mpiexec", which seems to be from the MPI installation used to
>> build PETSc. Probably the Paraview package installed its own MPI package.
>>
>>
>>>
>>> 2. The Makefile (as attached) which works fine before starts to complain:
>>> make: *** No rule to make target 'chkopts', needed by 'test'.  Stop.
>>>
>>
>> The 'chkopts' target is defined in the toplevel PETSc makefile, but it is
>> deprecated. You can take it out.
>>
>>   Thanks,
>>
>>  Matt
>>
>>
>>> Thanks for your kind help!
>>>
>>> Best regards,
>>> Kai
>>>
>>>
>>> PETSC_DIR=~/petsc
>>> PETSC_ARCH=arch-linux-c-debug
>>> CFLAGS = -I.
>>> FFLAGS=
>>> CPPFLAGS=-I.
>>> FPPFLAGS=
>>> LOCDIR=
>>> EXAMPLESC=
>>> EXAMPLESF=
>>> MANSEC=
>>> CLEANFILES=
>>> NP=
>>>
>>>
>>> include ${PETSC_DIR}/lib/petsc/conf/variables
>>> include ${PETSC_DIR}/lib/petsc/conf/rules
>>> include ${PETSC_DIR}/lib/petsc/conf/test
>>>
>>> test: ex2-2.o chkopts
>>> rm -rf topopt
>>> -${CLINKER} -o test ex2-2.o ${PETSC_SYS_LIB}
>>> ${RM} ex2-2.o
>>> rm -rf *.o
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] Something went wrong with PETSc after installing ParaView

[Matthew Knepley]

On Sat, Aug 5, 2023 at 3:22 AM K. Wu  wrote:

> Hi all,
>
> Good day!
>
> After installing ParaView on my desktop, PETSc starts to work anomalously
> even after reconfiguration:
> 1. If I use mpirun (frequently used before), it seems that now all the
> processors will run the program independently without communication. While
> mpiexec seems to work properly.
>

Yes, this indicates that the "mpirun" is from a different installation of
MPI than "mpiexec", which seems to be from the MPI installation used to
build PETSc. Probably the Paraview package installed its own MPI package.


>
> 2. The Makefile (as attached) which works fine before starts to complain:
> make: *** No rule to make target 'chkopts', needed by 'test'.  Stop.
>

The 'chkopts' target is defined in the toplevel PETSc makefile, but it is
deprecated. You can take it out.

  Thanks,

 Matt


> Thanks for your kind help!
>
> Best regards,
> Kai
>
>
> PETSC_DIR=~/petsc
> PETSC_ARCH=arch-linux-c-debug
> CFLAGS = -I.
> FFLAGS=
> CPPFLAGS=-I.
> FPPFLAGS=
> LOCDIR=
> EXAMPLESC=
> EXAMPLESF=
> MANSEC=
> CLEANFILES=
> NP=
>
>
> include ${PETSC_DIR}/lib/petsc/conf/variables
> include ${PETSC_DIR}/lib/petsc/conf/rules
> include ${PETSC_DIR}/lib/petsc/conf/test
>
> test: ex2-2.o chkopts
> rm -rf topopt
> -${CLINKER} -o test ex2-2.o ${PETSC_SYS_LIB}
> ${RM} ex2-2.o
> rm -rf *.o
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Setting a custom predictor in the generalized-alpha time stepper

[Matthew Knepley]

If you want to make a PR with your hack, we can help build out the
infrastructure for what Jed is recommending.

  Thanks,

 Matt

On Fri, Aug 4, 2023 at 2:56 PM Jed Brown  wrote:

> Yeah, we'd like the implementation to stay in alpha2.c. There could be a
> new interface TSAlpha2SetPredictorType (with -ts_alpha2_predictor_type
> [none,same_velocity,...]) or TSAlpha2SetPredictorFunction.
>
> David Kamensky  writes:
>
> > Hi Jed,
> >
> > The current workaround I'm using is very minimal and basically just moves
> > the definition of `TS_Alpha` from `alpha2.c` up to
> `petsc/private/tsimpl.h`
> > (and renames it to avoid a conflict with `TS_Alpha` in `alpha1.c`), but I
> > gather that we're really still not "supposed to" include that header in
> > applications.  So, I don't know whether something like that would be
> > welcomed upstream.  The actual computation of the predictor is done on
> the
> > application side.
> >
> > Having options like `-ts_alpha_same_velocity` and
> > `-ts_alpha_same_acceleration` could probably be implemented by analogy to
> > `-ts_theta_initial_guess_extrapolate`, although they wouldn't quite cover
> > my specific use-case, where I'm only setting the predictor on part of the
> > solution vector.  So, maybe something more general, like providing a
> > generalized-$\alpha$-specific option for a custom predictor callback that
> > takes `X0`, `V0`, and `A0` arguments would be the cleanest solution (and
> > some convenient shortcuts for full-solution same-velocity and
> > same-acceleration predictors could subsequently make use of that
> > infrastructure).  I've been working quickly over the past week, but I
> might
> > be able to take some time to implement a more sustainable solution soon.
> >
> > Thanks again,
> > David
> >
> > On Fri, Aug 4, 2023 at 9:23 AM Jed Brown  wrote:
> >
> >> Some other TS implementations have a concept of extrapolation as an
> >> initial guess. Such method-specific initial guesses sound like they fit
> >> that pattern and would be welcome to be included in alpha2.c. Would you
> be
> >> willing to make a merge request to bring your work upstream?
> >>
> >> David Kamensky  writes:
> >>
> >> > Hi Jed,
> >> >
> >> > What I'm trying to compute is basically a standard same-velocity or
> >> > same-acceleration predictor (although slightly more complicated, since
> >> I'm
> >> > restricting it to a sub-system).  I hadn't looked into
> >> > `SNESSetComputeInitialGuess` yet, although one difficulty is that it
> >> would
> >> > need access to the `X0`, `V0`, and `A0` members of the `TS_Alpha`
> struct,
> >> > which is only defined in `alpha2.c`, and thus not available through
> the
> >> > API.
> >> >
> >> > For now, we just worked around this by patching PETSc to move the
> >> > definition of `TS_Alpha` up into a header to make it accessible.
> >> > (Modifying the library obviously introduces a maintenance headache; I
> >> also
> >> > considered just casting the `ts->data` pointer to `(char*)`,
> calculating
> >> > memory offsets based on `sizeof` the struct members, and casting back
> to
> >> > `Vec`, but that relies on compiler-specific assumptions, and could
> also
> >> > break if the PETSc source code was updated.)  We also shuffled the
> order
> >> of
> >> > some calls to `VecCopy` and `TSPreStage` in the routine
> >> `TSAlpha_Restart`,
> >> > so that `TSPreStage` can set the initial guess, although that sounds
> like
> >> > it would be unnecessary if we instead used a callback in
> >> > `SNESSetComputeInitialGuess` that had access to the internals of
> >> > `TS_Alpha`.
> >> >
> >> > Thanks, David
> >> >
> >> > On Thu, Aug 3, 2023 at 11:28 PM Jed Brown  wrote:
> >> >
> >> >> I think you can use TSGetSNES() and SNESSetComputeInitialGuess() to
> >> modify
> >> >> the initial guess for SNES. Would that serve your needs? Is there
> >> anything
> >> >> else you can say about how you'd like to compute this initial guess?
> Is
> >> >> there a paper or something?
> >> >>
> >> >> David Kamensky  writes:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > My understanding is that the second-order generalized-alpha time
> >> stepper
> >> >> in
> >> >> > PETSc uses a same-displacement predictor as the initial guess for
> the
> >> >> > nonlinear solver that executes in each time step.  I'd like to be
> >> able to
> >> >> > set this to something else, to improve convergence.  However, my
> >> >> > (possibly-naive) attempts to use `TSSetPreStep` and `TSSetPreStage`
> >> >> haven't
> >> >> > worked out.  Is there any way to set a custom predictor?
> >> >> >
> >> >> > Thanks,
> >> >> > David Kamensky
> >> >>
> >>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] PETSc :: FEM Help

[Matthew Knepley]

On Fri, Aug 4, 2023 at 10:31 AM Brandon Denton  wrote:

> Good Morning Prof. Knepley,
>
> Thank you for the update. I am now able to run the code. However, it does
> not appear to solve the problem correctly. The only results available are
> the initial conditions (temp = 100). In the problem, one face is set to
> 1400 and another face is set to 100. Since the faces are at opposite ends
> of the geometry, we would expect a roughly linear temperature profile from
> 1400 to 100. What am I missing to get the output to show this proper result.
>

1) The inlet/outlet labels where being constructed on dmSurface. I fixed
this.

2) The 14/7 faceIDs do not appear to be the ones you want. Here is
corrected source. Can you look at the labels?

  Thanks,

 Matt


> Thank you.
> Brandon
>
>
> ------
> *From:* Matthew Knepley 
> *Sent:* Tuesday, August 1, 2023 10:23 AM
> *To:* Brandon Denton 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] PETSc :: FEM Help
>
> Sorry about this. I signed up for a conference without the work done, with
> predictable results. I have just returned home.
>
> There were just a few small problems. First, the labels were attached to
> dmSurface, but you wanted them on dm. They got destroyed with dmSurface
> before setting the BC. Second, the declarations of the point function were
> missing the constant arguments. Third, the PetscFEDestroy() was missing and
> extra DM creations were there. I have fixed these and am attaching the new
> source. It runs for me but I have not checked the answer.
>
>   Thanks,
>
>  Matt
>
> On Wed, Jun 7, 2023 at 11:05 AM Brandon Denton via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Good Morning,
>
> I'm trying to verify that the CAD -> PETSc/DMPlex methods I've developed
> can be used for FEM analyses using PETSc. Attached is my current attempt
> where I import a CAD STEP file to create a volumetric tetrahedral
> discretization (DMPlex),  designate boundary condition points using
> DMLabels, and solve the Laplace problem (heat) with Dirichlet conditions on
> each end. At command line I indicate the STEP file with the -filename
> option and the dual space degree with -petscspace_degree 2. The run ends
> with either a SEGV Fault or a General MPI Communication Error.
>
> Could you please look over the attached file to tell me if what I'm doing
> to set up the FEM problem is wrong?
>
> Thank you in advance for your time and help.
> -Brandon
>
> TYPICAL ERROR MESSAGE
> [0]PETSC ERROR: - Error Message
> --
> [0]PETSC ERROR: General MPI error
> [0]PETSC ERROR: MPI error 605109765 Invalid communicator, error stack:
> PMPI_Comm_get_attr(344): MPI_Comm_get_attr(comm=0x0,
> comm_keyval=-1539309568, attribute_val=0x7ffe75a58848, flag=0x7ffe75a58844)
> failed
> MPII_Comm_get_attr(257): MPIR_Comm_get_attr(comm=0x0,
> comm_keyval=-1539309568, attribute_val=0x7ffe75a58848, flag=0x7ffe75a58844)
> failed
> MPII_Comm_get_attr(53).: Invalid communicator
> [0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could
> be the program crashed before they were used or a spelling mistake, etc!
> [0]PETSC ERROR:   Option left: name:-dm_plex_refine_without_snap_to_geom
> value: 0 source: command line
> [0]PETSC ERROR:   Option left: name:-dm_refine value: 1 source: command
> line
> [0]PETSC ERROR:   Option left: name:-snes_monitor (no value) source:
> command line
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.18.5-1817-gd2497b8de4c
> GIT Date: 2023-05-22 18:44:03 +
> [0]PETSC ERROR: ./thermal on a  named XPS. by bdenton Wed Jun  7 11:03:43
> 2023
> [0]PETSC ERROR: Configure options --with-make-np=16
> --prefix=/mnt/c/Users/Brandon/software/libs/petsc/3.19.1-gitlab/gcc/11.2.0/mpich/3.4.2/openblas/0.3.17/opt
> --with-debugging=false --COPTFLAGS="-O3 -mavx" --CXXOPTFLAGS="-O3 -mavx"
> --FOPTFLAGS=-O3 --with-shared-libraries=1
> --with-mpi-dir=/mnt/c/Users/Brandon/software/libs/mpich/3.4.2/gcc/11.2.0
> --with-mumps=true --download-mumps=1 --with-metis=true --download-metis=1
> --with-parmetis=true --download-parmetis=1 --with-superlu=true
> --download-superlu=1 --with-superludir=true --download-superlu_dist=1
> --with-blacs=true --download-blacs=1 --with-scalapack=true
> --download-scalapack=1 --with-hypre=true --download-hypre=1
> --with-hdf5-dir=/mnt/c/Users/Brandon/software/libs/hdf5/1.12.1/gcc/11.2.0
> --with-valgrind-dir=/mnt/c/Users/Brandon/software/apps/valgrind/3.14.0
> --with-blas-lib=&qu

Re: [petsc-users] DMPlex edge/vertex orientation

[Matthew Knepley]

On Fri, Aug 4, 2023 at 12:06 PM onur.notonur via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hi,
>
> I'm currently working with 3D DMPlex and performing crucial calculations
> involving face normals and edge tangents. I've noticed that face normals
> are directed from support[0] to support[1].
>

That is an accident of implementation and not enforced.


> However, I'm uncertain about the conventions for edges and vertices in
> relation to faces. Specifically, I need to determine the order of vertices
> that create a surface and whether they are stored in a counter-clockwise
> (CCW) or clockwise (CW) manner. As DMPlex follows a hierarchy of
> cell-face-edge-vertex, my main question becomes about the orientation of
> edges. Any clarification on this aspect would be immensely helpful!
>

1) All computed quantities follow the closure ordering, namely that the
order that vertices come out in the DMPlexGetTransitiveClosure() call is
the one used for computing.

2) Closures are always ordered to produce outward normals

3) Since we build k-cells out of k-1 cells, the k-1 cells _already_ have an
ordering before I make my k-cell. Thus I have to tell you how to order the
k-1 cell, with respect to its closure ordering, when you are building your
k-cell. This is what an "orientation" is, namely a representation of the
dihedral group for that k-1 cell.

Example: A segment has two orientations, which we label 0 and -1. When we
build a triangle out of segments, we order them counter-clockwise, so that
the normals are all outward. The same thing is done
for quads.

Triangles have 6 orientations, all the permutations of the edges. We pick
one when making tetrahedra such that the normals are outward _and_ the
vertices are in the closure order.


> Additionally, I'm unfamiliar with most of the terms used in DMPlex. For
> example "orientation" in DMPlexGetConeOrientation. If you could suggest
> some readings or resources that explain these concepts, I would greatly
> appreciate it.
>

I am finishing up my book on it, which I will post. To start, here is a
paper

  https://arxiv.org/abs/2004.08729

  Thanks,

  Matt


> Thx,
> Onur
> Sent with Proton Mail  secure email.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] compiler related error (configuring Petsc)

[Matthew Knepley]

On Wed, Aug 2, 2023 at 9:32 AM maitri ksh  wrote:

> I could compile petsc using the newer version of gnu compiler.
> However, there is some unusual error when I tried to test petsc using a
> print 'hello' file. I could not interpret what the error ('*error.txt*')
> is, but it says something related to MATLAB (which is not used in the '
> *hello.c*' script). Any comments/suggestions?
>

1) Did you link PETSc against MATLAB? If not, did you link your test
against MATLAB?

2) If you linked PETSc against it, likely you were in a shell with
LD_LIBRARY_PATH defined so that MATLAB would run. Without that, it cannot
find the dynamic libraries it needs.

  Thanks,

 Matt


> On Wed, Aug 2, 2023 at 11:07 AM maitri ksh  wrote:
>
>> Okay, thank you.
>>
>> On Tue, Aug 1, 2023 at 9:43 PM Satish Balay  wrote:
>>
>>> > gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
>>>
>>> Is it possible for you to use a newer version GNU compilers?
>>>
>>> If not - your alternative is to build PETSc with --with-cxx=0 option
>>>
>>> But then - you can't use --download-superlu_dist or any pkgs that need
>>> c++ [you could try building them separately though]
>>>
>>> Satish
>>>
>>>
>>> On Tue, 1 Aug 2023, maitri ksh wrote:
>>>
>>> > I am trying to compile petsc on a cluster ( x86_64-redhat-linux, '
>>> > *configure.log'*  is attached herewith) . Initially I got an error
>>> related
>>> > to 'C++11' flag, to troubleshoot this issue, I used 'CPPFLAGS' and
>>> > 'CXXFLAGS' and could surpass the non-compliant error related to c++
>>> compiler
>>> >  but now it gives me another error 'cannot find a C preprocessor'. How
>>> to
>>> > fix this?
>>> >
>>>
>>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] PETSc :: FEM Help

[Matthew Knepley]

Sorry about this. I signed up for a conference without the work done, with
predictable results. I have just returned home.

There were just a few small problems. First, the labels were attached to
dmSurface, but you wanted them on dm. They got destroyed with dmSurface
before setting the BC. Second, the declarations of the point function were
missing the constant arguments. Third, the PetscFEDestroy() was missing and
extra DM creations were there. I have fixed these and am attaching the new
source. It runs for me but I have not checked the answer.

  Thanks,

 Matt

On Wed, Jun 7, 2023 at 11:05 AM Brandon Denton via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Good Morning,
>
> I'm trying to verify that the CAD -> PETSc/DMPlex methods I've developed
> can be used for FEM analyses using PETSc. Attached is my current attempt
> where I import a CAD STEP file to create a volumetric tetrahedral
> discretization (DMPlex),  designate boundary condition points using
> DMLabels, and solve the Laplace problem (heat) with Dirichlet conditions on
> each end. At command line I indicate the STEP file with the -filename
> option and the dual space degree with -petscspace_degree 2. The run ends
> with either a SEGV Fault or a General MPI Communication Error.
>
> Could you please look over the attached file to tell me if what I'm doing
> to set up the FEM problem is wrong?
>
> Thank you in advance for your time and help.
> -Brandon
>
> TYPICAL ERROR MESSAGE
> [0]PETSC ERROR: - Error Message
> --
> [0]PETSC ERROR: General MPI error
> [0]PETSC ERROR: MPI error 605109765 Invalid communicator, error stack:
> PMPI_Comm_get_attr(344): MPI_Comm_get_attr(comm=0x0,
> comm_keyval=-1539309568, attribute_val=0x7ffe75a58848, flag=0x7ffe75a58844)
> failed
> MPII_Comm_get_attr(257): MPIR_Comm_get_attr(comm=0x0,
> comm_keyval=-1539309568, attribute_val=0x7ffe75a58848, flag=0x7ffe75a58844)
> failed
> MPII_Comm_get_attr(53).: Invalid communicator
> [0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could
> be the program crashed before they were used or a spelling mistake, etc!
> [0]PETSC ERROR:   Option left: name:-dm_plex_refine_without_snap_to_geom
> value: 0 source: command line
> [0]PETSC ERROR:   Option left: name:-dm_refine value: 1 source: command
> line
> [0]PETSC ERROR:   Option left: name:-snes_monitor (no value) source:
> command line
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.18.5-1817-gd2497b8de4c
> GIT Date: 2023-05-22 18:44:03 +
> [0]PETSC ERROR: ./thermal on a  named XPS. by bdenton Wed Jun  7 11:03:43
> 2023
> [0]PETSC ERROR: Configure options --with-make-np=16
> --prefix=/mnt/c/Users/Brandon/software/libs/petsc/3.19.1-gitlab/gcc/11.2.0/mpich/3.4.2/openblas/0.3.17/opt
> --with-debugging=false --COPTFLAGS="-O3 -mavx" --CXXOPTFLAGS="-O3 -mavx"
> --FOPTFLAGS=-O3 --with-shared-libraries=1
> --with-mpi-dir=/mnt/c/Users/Brandon/software/libs/mpich/3.4.2/gcc/11.2.0
> --with-mumps=true --download-mumps=1 --with-metis=true --download-metis=1
> --with-parmetis=true --download-parmetis=1 --with-superlu=true
> --download-superlu=1 --with-superludir=true --download-superlu_dist=1
> --with-blacs=true --download-blacs=1 --with-scalapack=true
> --download-scalapack=1 --with-hypre=true --download-hypre=1
> --with-hdf5-dir=/mnt/c/Users/Brandon/software/libs/hdf5/1.12.1/gcc/11.2.0
> --with-valgrind-dir=/mnt/c/Users/Brandon/software/apps/valgrind/3.14.0
> --with-blas-lib="[/mnt/c/Users/Brandon/software/libs/openblas/0.3.17/gcc/11.2.0/lib/libopenblas.so]"
> --with-lapack-lib="[/mnt/c/Users/Brandon/software/libs/openblas/0.3.17/gcc/11.2.0/lib/libopenblas.so]"
> --LDFLAGS= --with-tetgen=true --download-tetgen=1 --download-ctetgen=1
> --download-opencascade=1 --download-egads
> [0]PETSC ERROR: #1 PetscObjectName() at
> /mnt/c/Users/Brandon/software/builddir/petsc-3.19.1-gitlab/src/sys/objects/pname.c:119
> [0]PETSC ERROR: #2 PetscObjectGetName() at
> /mnt/c/Users/Brandon/software/builddir/petsc-3.19.1-gitlab/src/sys/objects/pgname.c:27
> [0]PETSC ERROR: #3 PetscDSAddBoundary() at
> /mnt/c/Users/Brandon/software/builddir/petsc-3.19.1-gitlab/src/dm/dt/interface/dtds.c:3404
> [0]PETSC ERROR: #4 DMAddBoundary() at
> /mnt/c/Users/Brandon/software/builddir/petsc-3.19.1-gitlab/src/dm/interface/dm.c:7828
> [0]PETSC ERROR: #5 main() at
> /mnt/c/Users/Brandon/Documents/School/Dissertation/Software/EGADS-dev/thermal_v319/thermal_nozzle.c:173
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -dm_plex_geom_print_model 1 (source: command line)
> [0]PETSC ERROR: -dm_plex_geom_shape_opt 0 (source: command line)
> [0]PETSC ERROR: -dm_plex_refine_without_snap_to_geom 0 (source: command
> line)
> [0]PETSC ERROR: -dm_refine 1 (source: command line)
> [0]PETSC ERROR: -filename ./examples/Nozzle_example.stp (source: 

Re: [petsc-users] [petsc-maint] Monolithic AMG with fieldsplit as smoother

[Matthew Knepley]

On Thu, Jul 27, 2023 at 12:48 AM Jed Brown  wrote:

> AMG is subtle here. With AMG for systems, you typically feed it elements
> of the near null space. In the case of (smoothed) aggregation, the coarse
> space will have a regular block structure with block sizes equal to the
> number of near-null vectors. You can use pc_fieldsplit options to select
> which fields you want in each split.
>
> However, AMG also needs a strength of connection and if your system is so
> weird you need to fieldsplit the smoothers (e.g., a saddle point problem or
> a hyperbolic system) then it's likely that you'll also need a custom
> strength of connection to obtain reasonable coarsening.
>

For this reason, sometimes GMG is easier for systems since you just
rediscretize.

  Thanks,

 Matt


> Barry Smith  writes:
>
> >   See the very end of the section
> https://petsc.org/release/manual/ksp/#multigrid-preconditioners on how to
> control the smoothers (and coarse grid solve) for multigrid in PETSc
> including for algebraic multigrid.
> >
> >So, for example, -mg_levels_pc_type fieldsplit would be the starting
> point. Depending on the block size of the matrices it may automatically do
> simple splits, you can control the details  of the fieldsplit
> preconditioner with -mg_levels_pc_fieldsplit_...  and the details for each
> split with -mg_levels_fieldsplit_
> >
> >See src/ksp/ksp/tutorials/ex42.c for example, usage
> >
> >Feel free to ask more specific questions once you get started.
> >
> >> On Jul 26, 2023, at 9:47 PM, Michael Wick 
> wrote:
> >>
> >> Hello PETSc team:
> >>
> >> I wonder if the current PETSc implementation supports using AMG
> monolithically for a multi-field problem and using fieldsplit in the
> smoother.
> >>
> >> Thank you very much,
> >>
> >> Mike
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Inverse of a Large Sparse Matrix

[Matthew Knepley]

On Wed, Jul 26, 2023 at 8:13 AM maitri ksh  wrote:

> I have a large sparse matrix (48x48) and I need to take its
> inverse and use it to solve an eigenvalue problem. According to
> petsc-archive
> ,
> Barry suggested using  superLU with MatMatSolve() (& not KSPsolver) for
> matrix sizes of 5000 to 2. I was wondering two things:
> a) is it possible to avoid taking the explicit inverse of the large sparse
> matrix (as was discussed in the archive) for this particular case (in which
> I am using the matrix-inverse for an eigenvalue problem)
>

You do not actually want to explicitly invert in most instances. SLEPc will
do the right thing automatically.


> b) is KSPsolver more suitable here?
>

Most likely.


> Also, can someone please explain why SuperLU (SuperLU_SEQUENTIAL, which
> does not involve any parallel computation) is more efficient in dealing
> with large sparse matrices as compared to MATLAB's inbuilt LU
>  solver.
>

I have no idea what MATLAB is doing. However SuperLU uses the supernodal
formulation, and I am not sure that MATLAB does. If you care about the last
ounce of performance, it is always worth trying several packages, so you
might compare with the serial PETSc LU and MUMPS.

  Thanks,

 Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] UNABLE to read '.mat' file with 'PetscViewerHDF5Open' (WSL-Ubuntu22.04)

[Matthew Knepley]

On Tue, Jul 25, 2023 at 3:12 AM maitri ksh  wrote:

> Hi,
> I am new to Petsc, here are some details of the relevant softwares I am
> using:
> 1. petsc-3.19.3 (on wsl-ubuntu22.04 platform)
> 2. MATLAB-R2022a
> 3. hdf5-1.10.7 (checked using 'pkg-config --modversion hdf5')
> 4. configured using:  './configure --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack --with-matlab
> --with-matlab-dir=/usr/local/MATLAB/R2022a --download-hdf5 --with-hdf5=1'
>
> I am trying to read a '.mat' file ('*myfile.mat*') using
> 'PetscViewerHDF5Open'  (the source code '*read_Matrix.c*' is attached
> herewith). I could compile the source code with no error, but I am getting
> an error ('*ERROR.txt*') indicating that there is an issue opening the
> HDF5 file. Could you please help me to resolve this issue
>

HDF5 thinks that this is not an HDF5 file. Since PETSc built HDF5 for you,
you can check this using

  $PETSC_DIR/$PETSC_ARCH/bin/h5dump myfile.mat

It could be a version incompatibility between the HDF5 on your machine and
the one you asked PETSc to download.

  Thanks,

Matt


> Maitri
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] 3D Poisson solver failed in KSPSolve when number of cores is larger than one

[Matthew Knepley]

On Mon, Jul 24, 2023 at 8:16 PM Thuc Bui  wrote:

> Dear PETSc Users/Developers.
>
>
>
> I have been successfully using PETsc on Windows without MPI for a while
> now. I have now attempted to implement PETSc with MPI on Windows 10. I have
> built a release version of PETSc 3.18.6 with MS MPI 10.1.2, Intel MKL 3.279
> (2020) and Visual Studio 2019 as a static library. I am testing a finite
> difference 3D Poisson solver (cylindrical coordinates) with this MPI PETSc.
>
>
>
> On a Windows 10 laptop with 4 cores (8 logical processors), the solver
> works fine with one core. However, it gives errors with 2 cores. The errors
> are listed below followed by a complete code. The errors are generated in
> KSPSolve. I perhaps miss some settings to cause this error “No method
> muldiagonalblock for Mat of type seqaij”. I don’t know why for a 2-core MPI
> program that a seqaij matrix is used in KSPSolve.
>
>
>
> I would be very grateful if someone can provide me with some direction to
> fix these issues.
>

That is a bug with PCEISENSTAT. It is not frequently used. Since your
matrix does not use inodes, a default implementation of
MatMultDIagonalBlock is not found. We just need to add this for MATSEQAIJ.
It should work with a different PC. We will fix this as soon as we can.

  Thanks,

 Matt


> Many thanks for your help,
>
> Thuc Bui
>
> Senior R&D Engineer
>
> Calabazas Creek Research, Inc
>
> (650) 948-5361
>
>
>
>
>
> ///errors/
>
> rank=0 iStart=0 iEnd=2375
>
> rank=1 iStart=2375 iEnd=4750
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> - Error Message
> --
>
> - Error Message
> --
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> No support for this operation for this object type
>
> No support for this operation for this object type
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> No method multdiagonalblock for Mat of type seqaij
>
> No method multdiagonalblock for Mat of type seqaij
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> See https://petsc.org/release/faq/ for trouble shooting.
>
> See https://petsc.org/release/faq/ for trouble shooting.
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> Petsc Release Version 3.18.6, Mar 30, 2023
>
> Petsc Release Version 3.18.6, Mar 30, 2023
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24
> 15:16:55 2023
>
> [1]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc=0
> --with-cxx="win32fe cl" --with-debugging=0 --with-openmp
> --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include/x64
> --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpifec.lib]"
> --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_thread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/libiomp5md.lib]"
> --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/bin/x64/mpiexec
> --with-shared-libraries=0
>
> Poisson3D.exe on a v140-x64SReleaseSP named HERMES by bui Mon Jul 24
> 15:16:55 2023
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> #1 MatMultDiagonalBlock() at
> C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\INTERF~1\matrix.c:2538
>
> Configure options --with-cc="win32fe cl" --with-fc=0 --with-cxx="win32fe
> cl" --with-debugging=0 --with-openmp
> --with-mpi-include=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include:/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/include/x64
> --with-mpi-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpi.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/Lib/x64/msmpifec.lib]"
> --with-blaslapack-lib="[/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_lp64.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_core.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/mkl_intel_thread.lib,/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/IntelMKL/latest/x64/lib/libiomp5md.lib]"
> --with-mpiexec=/cygdrive/c/Users/bui/Documents/Petsc/externalpackages/MSMPI/latest/bin/x64/mpiexec
> --with-shared-libraries=0
>
> [1]PETSC ERROR:
>
> [0]PETSC ERROR:
>
> #2 MatMultDiagonalBlock_MPIAIJ() at
> C:\Users\bui\DOCUME~1\Petsc\latest\src\mat\impls\aij\mpi\mpiaij.c:978
>
> #1 MatMultDiag

Re: [petsc-users] support for mixed block size matrices/AIM in PETSc?

[Matthew Knepley]

On Mon, Jul 24, 2023 at 6:34 AM Daniel Stone 
wrote:

> Hello PETSc Users/Developers,
>
> A collegue of mine is looking into implementing an adaptive implicit
> method (AIM) over
> PETSc in our simulator. This has led to some interesting questions about
> what can
> be done with blocked matrices, which I'm not able to answer myself - does
> anyone have
> any insight?
>
> Apparently it would be ideal if we could find a matrix (and vector) type
> that supports a kind
> of mixed block size:
>
> "For AIM [...] we will have matrix elements of various shapes: 1x1, 1xN,
> Nx1 and NxN. [...]. The solution and residual will be a mix of 1 and N
> variable/cell block"
>

This is not the terminology we would use, since "blocksize" is usually
understood as something small
that is used for vectorization and indexing. These are large, O(N), and
require different implementation.
To build matrices out of these parts, you can use

  1) MatNest: Very small blocks are not ideal here, so

  2) MatLRC: This is for low-rank corrections, which is where small
matrices tend to arise

Does this make sense?

  Thanks,

Matt


> There are ideas for how to implement what we want using the
> fixed-block-size objects we
> understand well, but if anything like the above exists it would be very
> exciting.
>
>
> Thanks,
>
> Daniel
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Matrix assembly problem of overlapped DMPlex

[Matthew Knepley]

On Sun, Jul 23, 2023 at 11:54 AM 袁煕  wrote:

> 袁煕 
> 0:50 (0 分前)
> To PETSc
> Hi,
>
> I used PETSc to assemble a FEM stiff matrix of an overlapped (overlap=2)
> DMPlex and used the MUMPS solver to solve it. But I got a
> different solution by using 1 CPU and MPI parallel computation. I am
> wondering if I missed some necessary step or setting during my
> implementation.
>
> My calling process like follows
>
> 1. Generate matrix pkij from DMPlex dm_mesh
>
> ```
>   PetscCallA( DMCreateMatrix(dm_mesh, pkij, ierr) )
>   PetscCallA( MatSetBlockSize(pkij, 1, ierr))
>   if( overlap>0 ) call 
> MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr)
>   PetscCallA( MatSetFromOptions(pkij, ierr))
>
>   PetscCallA(MatGetLocalSize(pkij, m, n, ierr));
>   PetscCallA(MatGetSize(pkij, gm, gn, ierr));
>   PetscCallA(MatGetBlockSize(pkij, bs, ierr));
> ```
>
> 2. Do Matrix preallocation
>
> ```
>   PetscCallA(MatCreate(PETSC_COMM_WORLD, preallocator, ierr));
>   PetscCallA(MatSetType(preallocator, MATPREALLOCATOR, ierr));
>   PetscCallA(MatSetSizes(preallocator, m, n, gm, gn, ierr));
>   PetscCallA(MatSetBlockSize(preallocator, bs, ierr));
>   PetscCallA(MatSetUp(preallocator, ierr));
>
>   ...
>
>   PetscCallA( MatPreallocatorPreallocate(preallocator, PETSC_TRUE, pkij, 
> ierr) )
>   PetscCallA( MatDestroy(preallocator,ierr) )
> ```
>
> 3. Assemble matrix pkij by calling MatSetValue of all overlapped elements.
>
> In my above implementation, I used 
> MatSetOption(pkij,MAT_IGNORE_OFF_PROC_ENTRIES,PETSC_TRUE,ierr). Is that 
> correct? Or even other options are needed?
>
>
Do you want to ignore off-process entries? Usually in FEM we share part of
the boundary with other processes, and thus ignoring the off-process
entries will not give the correct result.

  Thanks,

 Matt


>
>
> Thanks for your help.
>
> --
>
> X. Yuan, Ph.D. of solid mechanics
>
> Advancesoft, Japan
>
> -
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] HELP: How to compile Fortran 2008 files using Petsc.

[Matthew Knepley]

On Sat, Jul 22, 2023 at 10:50 AM s.pradeep kumar 
wrote:

> Hi,
>
> Petsc by default seems to only recognize only the following files
>
> //
> .SUFFIXES: .F .F90 .f90 ${SUFFIXES} .PETSc .C .cc .cpp .cxx .r .rm .so
> .html .ad .m .F95 .f95 .fiat .cu .hip.cpp .kokkos.cxx .raja.cxx *.sycl.cxx
> //
>
> I want to compile .F08 files. How should I go about this?
>

The different suffixes are telling us to process them differently. I don't
think there is any difference in the processing for F90 and F08, so I would
just rename those files .F90.

  Thanks,

 Matt


> Regards,
> Pradeep
>
-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Python version of VecDuplicateVecs

[Matthew Knepley]

On Wed, Jul 19, 2023 at 7:59 AM Karsten Bolding <
kars...@bolding-bruggeman.com> wrote:

> Hello
>
> I'm re-implementing a model from mixed C/Fortran to Python where Petsc is
> used for matrix/vector operations.
>
> In the C-code I have:
> ierr = VecDuplicateVecs(templateVec,numTracers,&uef);
> to create uef with numTracers vectors.
>
> Later in the code looping is done of the individual vectors and a sequence
> of operations are performed.
>
> I would like to follow the same idea in Python - but do not find any
> example.
>
> An alternative would be to make a list of Vecs?
>

There is no optimization here, so I would just make a list of duplicates.

  Thanks,

 Matt


> Best Regards,
> Karsten
>
> --
> Karsten Bolding
> kars...@bolding-bruggeman.com
> +4564422058
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] periodic boundary conditions

[Matthew Knepley]

On Wed, Jul 19, 2023 at 6:23 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> I finally found the answer to my problem. I was not wrongly imposing the
> periodic boundary condition but rather was solving a problem (Poisson
> equation with PBC), which has an infinite family of solutions. I was
> recommended to add an artificial constraint or use a null-space-aware
> algebraic solver.
>
>
>
> Are MUMPS and Hypre null-space-aware algebraic solvers? How can I turn
> that option on?
>

MUMPS can do it with an option (it is in the documentation). For AMG you
need to specify the coarse solver. I know this works with GAMG, you use

  -mg_coarse_pc_type svd

  Thanks,

 Matt


> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Tuesday, 18 July 2023 at 17:22
> *To: *Barry Smith 
> *Cc: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>, petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject: *Re: [petsc-users] periodic boundary conditions
>
> Jed creates the LocalToGlobal that does this elimination in plexsfc.c
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> On Tue, Jul 18, 2023 at 12:07 PM Barry Smith  wrote:
>
>
>
>They are never really "eliminated" because extra copies in the global
> vector never exist.
>
>
>
> On Jul 18, 2023, at 12:03 PM, Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
>
>
> Thank you, Barry. I am using the MPIAIJ format for a Finite Element
> application. So, I am trying to understand what is implemented in DMDA to
> eliminate those extra nodes.
>
>
>
> Best,
>
> Karthik.
>
>
>
> *From: *Barry Smith 
> *Date: *Tuesday, 18 July 2023 at 16:58
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *Matthew Knepley , petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject: *Re: [petsc-users] periodic boundary conditions
>
>
>
>If you are using DMDA with periodic boundary conditions for example
> only one "copy" of such nodes exists in the global vector (the vector the
> solvers see) so one does not need to eliminate extra ones
>
>
>
> On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
>
>
> Yes, I clearly understand I need to eliminate one set of periodic nodes. I
> was hoping to use x = P x’ to eliminate one set. It is a kind of mapping.
>
>
>
> Sorry, I am not sure if it is the LocalToGlobal mapping you are referring
> to. Is there an example or reference to show how the LocalToGlobal mapping
> is being used to impose PBC?
>
>
>
> Best,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Tuesday, 18 July 2023 at 16:38
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] periodic boundary conditions
>
> On Tue, Jul 18, 2023 at 11:18 AM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> Thanks Matt.
>
>
>
> The mesh is structured (rectilinear), so it is periodic in that sense.
>
>
>
> Can you please explain how I can impose it strongly?
>
>
>
> Strongly means make those variables equal in a pointwise sense. We do this
> in the
>
> LocalToGlobal mapping, so one set is eliminated in the global problem.
>
>
>
>   Thanks,
>
>
>
> Matt
>
>
>
> My initial thought was to come up with a relation between the periodic
> nodes:
>
>
>
> x = P x’
>
>
>
> Say for 1-D problem with two elements
>
> (1)-(2)(3)
>
>
>
> P = [1 0, 0 1, 1 0]
>
> x = [x1 x2 x3]
>
> x’ = [x1 x2]
>
>
>
> and solve
>
> [P^T A P] x’ = P^T b
>
>
>
> I don’t think [P^T A P] is deterministic.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Tuesday, 18 July 2023 at 14:31
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] periodic boundary conditions
>
> On Tue, Jul 18, 2023 at 9:02 AM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
>
>
> This is exactly not a PETSc question. I am solving a Poisson equation
> using finite elements. I would like to impose PBC. I am thinking of using
> the Lagrange multiplier method

Re: [petsc-users] periodic boundary conditions

[Matthew Knepley]

Jed creates the LocalToGlobal that does this elimination in plexsfc.c

  Thanks,

 Matt

On Tue, Jul 18, 2023 at 12:07 PM Barry Smith  wrote:

>
>They are never really "eliminated" because extra copies in the global
> vector never exist.
>
> On Jul 18, 2023, at 12:03 PM, Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> Thank you, Barry. I am using the MPIAIJ format for a Finite Element
> application. So, I am trying to understand what is implemented in DMDA to
> eliminate those extra nodes.
>
> Best,
> Karthik.
>
>
> *From: *Barry Smith 
> *Date: *Tuesday, 18 July 2023 at 16:58
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *Matthew Knepley , petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject: *Re: [petsc-users] periodic boundary conditions
>
>If you are using DMDA with periodic boundary conditions for example
> only one "copy" of such nodes exists in the global vector (the vector the
> solvers see) so one does not need to eliminate extra ones
>
>
> On Jul 18, 2023, at 11:51 AM, Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Yes, I clearly understand I need to eliminate one set of periodic nodes. I
> was hoping to use x = P x’ to eliminate one set. It is a kind of mapping.
>
> Sorry, I am not sure if it is the LocalToGlobal mapping you are referring
> to. Is there an example or reference to show how the LocalToGlobal
> mapping is being used to impose PBC?
>
> Best,
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Tuesday, 18 July 2023 at 16:38
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] periodic boundary conditions
> On Tue, Jul 18, 2023 at 11:18 AM Karthikeyan Chockalingam - STFC UKRI <
> karthikeyan.chockalin...@stfc.ac.uk> wrote:
>
> Thanks Matt.
>
> The mesh is structured (rectilinear), so it is periodic in that sense.
>
> Can you please explain how I can impose it strongly?
>
>
> Strongly means make those variables equal in a pointwise sense. We do this
> in the
> LocalToGlobal mapping, so one set is eliminated in the global problem.
>
>   Thanks,
>
> Matt
>
>
> My initial thought was to come up with a relation between the periodic
> nodes:
>
> x = P x’
>
> Say for 1-D problem with two elements
> (1)-(2)(3)
>
> P = [1 0, 0 1, 1 0]
> x = [x1 x2 x3]
> x’ = [x1 x2]
>
> and solve
> [P^T A P] x’ = P^T b
>
> I don’t think [P^T A P] is deterministic.
>
> Kind regards,
> Karthik.
>
>
>
> *From: *Matthew Knepley 
> *Date: *Tuesday, 18 July 2023 at 14:31
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] periodic boundary conditions
> On Tue, Jul 18, 2023 at 9:02 AM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
> This is exactly not a PETSc question. I am solving a Poisson equation
> using finite elements. I would like to impose PBC. I am thinking of using
> the Lagrange multiplier method to impose them as constraints. Or do you
> think I could take an alternative approach?
>
>
> There are several options:
>
> 1) Just make a periodic mesh. This is what Plex does by default.
>
> 2) Impose the conditions strongly. This is what is done if you create the
> ZBox shape in Plex.
>
> 3) Impose the conditions weakly. This is what you are doing with Lagrange
> multipliers. You could
> also do a Nitsche boundary condition for this.
>
> Since the constraint is so simple, I do not see an advantage to imposing
> it weakly.
>
>   Thanks,
>
>  Matt
>
>
> Thank you for your help.
>
> Kind regards,
> Karthik.
>
> --
> *Dr. Karthik Chockalingam*
> High Performance Software Engineering Group
> Hartree Centre | Science and Technology Facilities Council
> karthikeyan.chockalin...@stfc.ac.uk
>
>  
>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] periodic boundary conditions

[Matthew Knepley]

On Tue, Jul 18, 2023 at 11:18 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalin...@stfc.ac.uk> wrote:

> Thanks Matt.
>
>
>
> The mesh is structured (rectilinear), so it is periodic in that sense.
>
>
>
> Can you please explain how I can impose it strongly?
>

Strongly means make those variables equal in a pointwise sense. We do this
in the
LocalToGlobal mapping, so one set is eliminated in the global problem.

  Thanks,

Matt


> My initial thought was to come up with a relation between the periodic
> nodes:
>
>
>
> x = P x’
>
>
>
> Say for 1-D problem with two elements
>
> (1)-(2)(3)
>
>
>
> P = [1 0, 0 1, 1 0]
>
> x = [x1 x2 x3]
>
> x’ = [x1 x2]
>
>
>
> and solve
>
> [P^T A P] x’ = P^T b
>
>
>
> I don’t think [P^T A P] is deterministic.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
>
>
> *From: *Matthew Knepley 
> *Date: *Tuesday, 18 July 2023 at 14:31
> *To: *Chockalingam, Karthikeyan (STFC,DL,HC) <
> karthikeyan.chockalin...@stfc.ac.uk>
> *Cc: *petsc-users@mcs.anl.gov 
> *Subject: *Re: [petsc-users] periodic boundary conditions
>
> On Tue, Jul 18, 2023 at 9:02 AM Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
>
>
> This is exactly not a PETSc question. I am solving a Poisson equation
> using finite elements. I would like to impose PBC. I am thinking of using
> the Lagrange multiplier method to impose them as constraints. Or do you
> think I could take an alternative approach?
>
>
>
> There are several options:
>
>
>
> 1) Just make a periodic mesh. This is what Plex does by default.
>
>
>
> 2) Impose the conditions strongly. This is what is done if you create the
> ZBox shape in Plex.
>
>
>
> 3) Impose the conditions weakly. This is what you are doing with Lagrange
> multipliers. You could
>
> also do a Nitsche boundary condition for this.
>
>
>
> Since the constraint is so simple, I do not see an advantage to imposing
> it weakly.
>
>
>
>   Thanks,
>
>
>
>  Matt
>
>
>
> Thank you for your help.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
> --
>
> *Dr. Karthik Chockalingam*
>
> High Performance Software Engineering Group
>
> Hartree Centre | Science and Technology Facilities Council
>
> karthikeyan.chockalin...@stfc.ac.uk
>
>
>
>  [image: signature_3970890138]
>
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

Re: [petsc-users] periodic boundary conditions

[Matthew Knepley]

On Tue, Jul 18, 2023 at 9:02 AM Karthikeyan Chockalingam - STFC UKRI via
petsc-users  wrote:

> Hello,
>
>
>
> This is exactly not a PETSc question. I am solving a Poisson equation
> using finite elements. I would like to impose PBC. I am thinking of using
> the Lagrange multiplier method to impose them as constraints. Or do you
> think I could take an alternative approach?
>

There are several options:

1) Just make a periodic mesh. This is what Plex does by default.

2) Impose the conditions strongly. This is what is done if you create the
ZBox shape in Plex.

3) Impose the conditions weakly. This is what you are doing with Lagrange
multipliers. You could
also do a Nitsche boundary condition for this.

Since the constraint is so simple, I do not see an advantage to imposing it
weakly.

  Thanks,

 Matt


> Thank you for your help.
>
>
>
> Kind regards,
>
> Karthik.
>
>
>
> --
>
> *Dr. Karthik Chockalingam*
>
> High Performance Software Engineering Group
>
> Hartree Centre | Science and Technology Facilities Council
>
> karthikeyan.chockalin...@stfc.ac.uk
>
>
>
>  [image: signature_3970890138]
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Bug Report TaoALMM class

[Matthew Knepley]

Toby and Hansol,

Has anyone looked at this?

  Thanks,

 Matt

On Mon, Jun 12, 2023 at 8:24 AM Stephan Köhler <
stephan.koeh...@math.tu-freiberg.de> wrote:

> Dear PETSc/Tao team,
>
> I think there might be a bug in the Tao ALMM class:  In the function
> TaoALMMComputeAugLagAndGradient_Private(), see, eg.
>
> https://petsc.org/release/src/tao/constrained/impls/almm/almm.c.html#TAOALMM
> line 648 the gradient seems to be wrong.
>
> The given function and gradient computation is
> Lc = F + Ye^TCe + Yi^T(Ci - S) + 0.5*mu*[Ce^TCe + (Ci - S)^T(Ci - S)],
> dLc/dX = dF/dX + Ye^TAe + Yi^TAi + 0.5*mu*[Ce^TAe + (Ci - S)^TAi],
>
> but I think the gradient should be (without 0.5)
>
> dLc/dX = dF/dX + Ye^TAe + Yi^TAi + mu*[Ce^TAe + (Ci - S)^TAi].
>
> Kind regards,
> Stephan Köhler
>
> --
> Stephan Köhler
> TU Bergakademie Freiberg
> Institut für numerische Mathematik und Optimierung
>
> Akademiestraße 6
> 09599 Freiberg
> Gebäudeteil Mittelbau, Zimmer 2.07
>
> Telefon: +49 (0)3731 39-3173 (Büro)
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Inquiry about reading the P2 tetrahedron mesh from GMSH

[Matthew Knepley]

On Fri, Jun 30, 2023 at 4:40 PM neil liu  wrote:

> Dear Petsc developers,
>
> I am reading P2 mesh from GMSH. And used DMFieldGetClosure_Internal to
> check the coordinates for each tetrahedron, It seems reasonable.
> But when I tried DMGetCoordinates (dm, &global),  it seems the vector
> global is not consistent with the node number, Then what is global here?
>

Reading high order coordinates works for quads/hexes, but not simplices
right now because I need to figure out what order GMsh stores the
coordinates in. Thus we are defaulting to affine coordinates here.

  Thanks,

 Matt


> Thanks,
>
> Xiaodong
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] [EXTERNAL] PETSc Installation Assistance

[Matthew Knepley]

Hi Jesus,

I think you are on main. Did everything you have get rebuilt? Toby just
rewrote large sections of logging
and this is right where it fails for you. It should be easy to see what is
wrong by running in the debugger.

  Thanks,

 Matt

On Mon, Jul 17, 2023 at 3:11 PM Pierre Jolivet 
wrote:

>
> On 17 Jul 2023, at 9:00 PM, Ferrand, Jesus A. 
> wrote:
>
> Pierre:
> Setting the environment variable allows make check to complete without
> errors.
> It only seems to run three checks.
> I recall that in the past, make check would run 100+ tests.
>
>
> You are probably thinking of make test, which is running thousands of
> tests indeed.
>
> This is what I get from make check:
> =
> Running check examples to verify correct installation
> Using PETSC_DIR=/home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc and
> PETSC_ARCH=arch-linux-c-debug
> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
>
> C/C++ example src/vec/vec/tests/ex47 run successfully with hdf5
> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process
>
> Completed test examples
> =
>
> Anyhow, some of the APIs that worked before now break.
> For example "DMPlexCreateGmshFromFile()" breaks when I read simple test
> mesh.
> I tried configuring and installing both the newest release of PETSc and
> that tarball of version 3.18.3.
> Both seem to be breaking APIs that used to work in the older OS.
>
>
> There has been some internal changes to the behavior of DMPlex, they
> should be documented in the change log.
> But you are saying that with the same version, just by switching OS, you
> get a different behavior?
> Do as Satish suggested then, because it’s unlikely that the root cause is
> in the library itself.
>
> Thanks,
> Pierre
>
> =
> [0]PETSC ERROR:
> 
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and
> https://petsc.org/release/faq/
> [0]PETSC ERROR: -  Stack Frames
> 
> [0]PETSC ERROR: The line numbers in the error traceback are not always
> exact.
> [0]PETSC ERROR: #1 DMPlexSetMigrationSF() at
> /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/ple
> xnatural.c:21
> [0]PETSC ERROR: #2 JAF_DMPlexCreateFromMesh() at
> /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5865
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59
> message=application called MPI_Abort(MPI_COMM_WORLD, 59)
> - process 0
> :
> system msg for write_line failure : Bad file descriptor
> [0]PETSC ERROR:
> 
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and
> https://petsc.org/release/faq/
> [0]PETSC ERROR: -  Stack Frames
> 
> [0]PETSC ERROR: The line numbers in the error traceback are not always
> exact.
> [0]PETSC ERROR: #1 DMPlexSetMigrationSF() at
> /home/jesus/Desktop/JAF_NML/3rd_Party/PETSc/petsc/src/dm/impls/plex/ple
> xnatural.c:21
> [0]PETSC ERROR: #2 JAF_DMPlexCreateFromMesh() at
> /home/jesus/Desktop/JAF_NML/ApplicationCode/PETSc/PETScCGH5.c:5865
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [unset]: PMIU_write error; fd=-1 buf=:cmd=abort exitcode=59
> message=application called MPI_Abort(MPI_COMM_WORLD, 59)
> - process 0
> :
> system msg for write_line failure : Bad file descriptor
> --
>
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code. Per user-direction, the job has been aborted.
> --
>
> [0]PETSC ERROR:
> 
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the
> batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
>
> [0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and
> https://petsc.org/release/faq/
> [0]PETSC ERROR: -  Stack Frames
> 
> [0]PE

Re: [petsc-users] Parallel matrix multiplication

[Matthew Knepley]

On Mon, Jul 17, 2023 at 2:22 PM Barry Smith  wrote:

>
>
>https://petsc.org/release/manualpages/Mat/MatPtAP/  also note that
> PETSc has a large infrastructure for efficient ways to compute various
> matrix-matrix operations with a variety of algorithms that can be all
> accessed by starting with
> https://petsc.org/release/manualpages/Mat/MatProductCreate/
>
>
> On Jul 17, 2023, at 1:13 PM, Karthikeyan Chockalingam - STFC UKRI via
> petsc-users  wrote:
>
> Hello,
>
> I would like to perform the following operation
>
> [P^T A P] x’ = P^T f
>
> If P is thin, you might want to consider making a MatLRC and passing that
to a Krylov method, probably with a small precondition matrix (like P^T
diag(A) P).

  Thanks,

 Matt


> Where P is a rectangular matrix and A is a square matrix.  All the
> matrixes are constructed using MPIAIJ.
>
> Should I be concerned about the parallel partitioning of the matrix P and
> A?
>
> Or can I just go ahead and use MatMatMult to compute [P^T A P]?
>
> Is there any in-built functionality to perform this operation?
>
> Kind regards,
> Karthik.
>
>
>
>
>
>
>
> --
> *Dr. Karthik Chockalingam*
> High-Performance Software Engineer
> Hartree Centre | Science and Technology Facilities Council
> karthikeyan.chockalin...@stfc.ac.uk
>
>  
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Structured (DMDA) vs Unstructured (DMPlex) meshes

[Matthew Knepley]

On Mon, Jul 17, 2023 at 12:48 PM Barry Smith  wrote:

>
>The largest potential advantage of DMDA is likely the possibility of
> easily using geometric multigrid if it is appropriate for the problem (or
> subproblem of the problem) you are solving. The second advantage is, this
> depends on your PDE and discretization, the simplicity of your code, and
> what part of PETSc it depends on, if it doesn't depend on DMPLEX (and
> doesn't need to) then that is a good thing. Disadvantage of DMDA  is lack
> of flexibility in discretization, handling of non-standard boundary
> conditions, geometry.
>

I will be more specific about the discretizations. DMDA supports collocated
discretizations. If you have this, I would definitely recommend it, since
the array programming style is easy. However, if you have mixed FEM, or
higher order, or staggered discretizations, or complex BC, you need
something else.

There should be no difference in the solver since all the DMs just build
Vec and Mat objects and hand them off. There is a potential difference for
matrix-free operation. Right now, Plex is slow there, so we are putting in
an interface to libCEED for that.

  Thanks,

 Matt


> > On Jul 17, 2023, at 12:42 PM, Miguel Angel Salazar de Troya <
> miguel.sala...@corintis.com> wrote:
> >
> > Hello,
> >
> > I am trying to understand if I should make the effort to make my code
> use structured meshes instead of unstructured ones. My domain is cartesian
> so that is the first check for structured meshes. However, the problem size
> I am looking at is ~20 million degrees of freedom. My understanding is that
> for this problem size, most of the time is spent on the solver. In this
> case, do structured meshes still have an advantage? Can they run Krylov
> methods faster than when using structured meshes? What about other solvers
> and preconditioners?
> >
> > Thanks,
> > Miguel
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] petscconf.h missing building cpp file with dolfinx?

[Matthew Knepley]

On Mon, Jul 17, 2023 at 10:55 AM philliprusso via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> I cloned petsc source for use with dolfinx project. So after .configure
> mak sudo make install I found there was some type of difficulty with the
> destination directory so I copied the files manually into usr/includes of
> Ubuntu 22.04 jammy. So some petsc header files are now found for compiling
> cpp dolfinx source code but petscconf.h still not found by dolfinxs source
> tree of header files. Anyone know how to remedy this so dolfinx cpps can
> find petsscconf.h that g++ is claiming as missing? Thank you!
>

1. I think the safest thing is to fix the install

2. petscconf.h is in $PETSC_ARCH/include, not the top-level include, since
it is build-specific

  Thanks,

  Matt


> Sent with Proton Mail  secure email.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] windows build

[Matthew Knepley]

On Mon, Jul 17, 2023 at 6:08 AM Константин via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello. I have such problem, while I'm making petsc
> $ make
> C:/cygwin64/home/itugr/asd/petsc//lib/petsc/conf/petscvariables:140: ***
> Too many open files.  Stop.
>  Are there any advices how to fix it?
>

This is an OS problem, and since this is Windows, I would try rebooting.

  THanks,

Matt


> --
> Константин
>
>
>
> Вторник, 11 июля 2023, 23:09 +03:00 от Satish Balay :
>
> On Tue, 11 Jul 2023, Константин via petsc-users wrote:
>
> >
> > Hello, I'm trying to build petsc on windows. And when I make it I have
> such problem
>
> from the screenshot - it appears you are not using cygwin
> terminal(bash-shell) - as per instructions
>
> https://petsc.org/release/install/windows/
>
> What compilers are you attempting to use with PETSc? Can you retry your
> build - as per the above instructions?
>
> If you still encounter issues - send us the corresponding configure.log.
>
> Also - best if you can copy/paste text from terminal - instead of
> screenshots
>
> Note: if you can use WSL2(linux) on your windows machine - that might be
> an easier install
>
> Satish
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Bug in log_view printing

[Matthew Knepley]

On Sun, Jul 16, 2023 at 11:45 AM Jared Crean  wrote:

> Hello,
>
>I stumbled upon a small bug in the printing of the data when running
> Petsc with -log_view.  The entire table is attached, but here is an
> example row:
>

Yes, we prescribe the exact number of spaces for each output, instead of
allowing it to grow,
so if the number exceeds the size, we lose our space. We wanted to keep the
line length fixed.
We do not recommend this form for processing. We usually use either the XML
output for flame
graphs since it does nesting, or the CSV output for processing with Pandas.

  Thanks,

 Matt


>
> 
> EventCount  Time (sec)
> Flop  --- Global ---  --- Stage  Total
> Max Ratio  Max Ratio   Max  Ratio  Mess AvgLen
> Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>
> 
>
> --- Event Stage 0: Main Stage
>
> BuildTwoSided  7 1.0 1.3282e-0271.6 0.00e+00 0.0 6.0e+01 4.0e+00
> 5.0e+00  0  0  0  0  0   0  0  0  0  0 0
>
>
>Notice how the Time Max and Time Ratio values are missing a space
> separator (the other rows have only two digits for the exponents, I
> think there should be a space after the e-02 and before the 7).
>
>Configuration details:
>
>  * Petsc 3.17
>
>  * Arm64 architecture (and AWS c7g.xlarge instance, which use the
> new Graviton3 processor)
>
>  * ubuntu 22.04
>
>  * MPICH MPI (version 4.0, I think, whatever apt-get install mpich
> gives)
>
>  * clang-15 compiler
>
>
>Jared Crean
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Using PETSc GPU backend

[Matthew Knepley]

On Sat, Jul 15, 2023 at 1:44 AM Ng, Cho-Kuen  wrote:

> Matt,
>
> After inserting 2 lines in the code:
>
>   ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
>
>   ierr = MatSetFromOptions(A);CHKERRQ(ierr);
>   ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
>   d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
>
> "There are no unused options." However, there is no improvement on the GPU
> performance.
>

1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you
created in steps 1 and 2. This is detailed in the manual.

2. You should replace MatCreateAIJ(), with MatSetSizes() before
MatSetFromOptions().

  THanks,

Matt


> Thanks,
> Cho
>
> --
> *From:* Matthew Knepley 
> *Sent:* Friday, July 14, 2023 5:57 PM
> *To:* Ng, Cho-Kuen 
> *Cc:* Barry Smith ; Mark Adams ;
> petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen 
> wrote:
>
> I managed to pass the following options to PETSc using a GPU node on
> Perlmutter.
>
> -mat_type aijcusparse -vec_type cuda -log_view -options_left
>
> Below is a summary of the test using 4 MPI tasks and 1 GPU per task.
>
> o #PETSc Option Table entries:
> -log_view
> -mat_type aijcusparse
>-options_left
>-vec_type cuda
>#End of PETSc Option Table entries
>WARNING! There are options you set that were not used!
>WARNING! could be spelling mistake, etc!
>There is one unused database option. It is:
>Option left: name:-mat_type value: aijcusparse
>
> The -mat_type option has not been used. In the application code, we use
>
> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
>  d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
>
>
> If you create the Mat this way, then you need MatSetFromOptions() in order
> to set the type from the command line.
>
>   Thanks,
>
>  Matt
>
>
> o The percent flops on the GPU for KSPSolve is 17%.
>
> In comparison with a CPU run using 16 MPI tasks, the GPU run is an order
> of magnitude slower. How can I improve the GPU performance?
>
> Thanks,
> Cho
> --
> *From:* Ng, Cho-Kuen 
> *Sent:* Friday, June 30, 2023 7:57 AM
> *To:* Barry Smith ; Mark Adams 
> *Cc:* Matthew Knepley ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Barry, Mark and Matt,
>
> Thank you all for the suggestions. I will modify the code so we can pass
> runtime options.
>
> Cho
> --
> *From:* Barry Smith 
> *Sent:* Friday, June 30, 2023 7:01 AM
> *To:* Mark Adams 
> *Cc:* Matthew Knepley ; Ng, Cho-Kuen <
> c...@slac.stanford.edu>; petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
>
>   Note that options like -mat_type aijcusparse  -vec_type cuda only work
> if the program is set up to allow runtime swapping of matrix and vector
> types. If you have a call to MatCreateMPIAIJ() or other specific types then
> then these options do nothing but because Mark had you use -options_left
> the program will tell you at the end that it did not use the option so you
> will know.
>
> On Jun 30, 2023, at 9:30 AM, Mark Adams  wrote:
>
> PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args
> and you run:
>
> a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left
>
> Mark
>
> On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley  wrote:
>
> On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Mark,
>
> The application code reads in parameters from an input file, where we can
> put the PETSc runtime options. Then we pass the options to
> PetscInitialize(...). Does that sounds right?
>
>
> PETSc will read command line argument automatically in PetscInitialize()
> unless you shut it off.
>
>   Thanks,
>
> Matt
>
>
> Cho
> --
> *From:* Ng, Cho-Kuen 
> *Sent:* Thursday, June 29, 2023 8:32 PM
> *To:* Mark Adams 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Mark,
>
> Thanks for the information. How do I put the runtime options for the
> executable, say, a.out, which does not have the provision to append
> arguments? Do I need to change the C++ main to read in the options?
>
> Cho
> --
> *From:* Mark Adams 
> *Sent:* Thursday, June 29, 2023 5:55 PM
> *To:* Ng, Cho-Kuen 
> *Cc:* petsc-users@mc

Re: [petsc-users] Using PETSc GPU backend

[Matthew Knepley]

On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen  wrote:

> I managed to pass the following options to PETSc using a GPU node on
> Perlmutter.
>
> -mat_type aijcusparse -vec_type cuda -log_view -options_left
>
> Below is a summary of the test using 4 MPI tasks and 1 GPU per task.
>
> o #PETSc Option Table entries:
> -log_view
> -mat_type aijcusparse
>-options_left
>-vec_type cuda
>#End of PETSc Option Table entries
>WARNING! There are options you set that were not used!
>WARNING! could be spelling mistake, etc!
>There is one unused database option. It is:
>Option left: name:-mat_type value: aijcusparse
>
> The -mat_type option has not been used. In the application code, we use
>
> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,
>  d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);
>
>
If you create the Mat this way, then you need MatSetFromOptions() in order
to set the type from the command line.

  Thanks,

 Matt


> o The percent flops on the GPU for KSPSolve is 17%.
>
> In comparison with a CPU run using 16 MPI tasks, the GPU run is an order
> of magnitude slower. How can I improve the GPU performance?
>
> Thanks,
> Cho
> --
> *From:* Ng, Cho-Kuen 
> *Sent:* Friday, June 30, 2023 7:57 AM
> *To:* Barry Smith ; Mark Adams 
> *Cc:* Matthew Knepley ; petsc-users@mcs.anl.gov <
> petsc-users@mcs.anl.gov>
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Barry, Mark and Matt,
>
> Thank you all for the suggestions. I will modify the code so we can pass
> runtime options.
>
> Cho
> --
> *From:* Barry Smith 
> *Sent:* Friday, June 30, 2023 7:01 AM
> *To:* Mark Adams 
> *Cc:* Matthew Knepley ; Ng, Cho-Kuen <
> c...@slac.stanford.edu>; petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
>
>   Note that options like -mat_type aijcusparse  -vec_type cuda only work
> if the program is set up to allow runtime swapping of matrix and vector
> types. If you have a call to MatCreateMPIAIJ() or other specific types then
> then these options do nothing but because Mark had you use -options_left
> the program will tell you at the end that it did not use the option so you
> will know.
>
> On Jun 30, 2023, at 9:30 AM, Mark Adams  wrote:
>
> PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args
> and you run:
>
> a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left
>
> Mark
>
> On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley  wrote:
>
> On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> Mark,
>
> The application code reads in parameters from an input file, where we can
> put the PETSc runtime options. Then we pass the options to
> PetscInitialize(...). Does that sounds right?
>
>
> PETSc will read command line argument automatically in PetscInitialize()
> unless you shut it off.
>
>   Thanks,
>
> Matt
>
>
> Cho
> --
> *From:* Ng, Cho-Kuen 
> *Sent:* Thursday, June 29, 2023 8:32 PM
> *To:* Mark Adams 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Mark,
>
> Thanks for the information. How do I put the runtime options for the
> executable, say, a.out, which does not have the provision to append
> arguments? Do I need to change the C++ main to read in the options?
>
> Cho
> --
> *From:* Mark Adams 
> *Sent:* Thursday, June 29, 2023 5:55 PM
> *To:* Ng, Cho-Kuen 
> *Cc:* petsc-users@mcs.anl.gov 
> *Subject:* Re: [petsc-users] Using PETSc GPU backend
>
> Run with options: -mat_type aijcusparse -vec_type cuda -log_view
> -options_left
>
> The last column of the performance data (from -log_view) will be the
> percent flops on the GPU. Check that that is > 0.
>
> The end of the output will list the options that were used and options
> that were _not_ used (if any). Check that there are no options left.
>
> Mark
>
> On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
>
> I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and
> used it by "spack load petsc/fwge6pf". Then I compiled the application
> code (purely CPU code) linking to the petsc package, hoping that I can get
> performance improvement using the petsc GPU backend. However, the timing
> was the same using the same number of MPI tasks with and without GPU
> accelerators. Have I missed something in the process, for example, setting
&

Re: [petsc-users] Near null space for a fieldsplit in petsc4py

[Matthew Knepley]

On Thu, Jul 13, 2023 at 5:33 AM Pierre Jolivet 
wrote:

> Dear Nicolas,
>
> On 13 Jul 2023, at 10:17 AM, TARDIEU Nicolas 
> wrote:
>
> Dear Pierre,
>
> You are absolutely right. I was using a --with-debugging=0 (aka release)
> install and this is definitely an error.
> Once I used my debug install, I found the way to fix my problem. The
> solution is in the attached script: I first need to extract the correct
> block from the PC operator's MatNest and then append the null space to it.
> Anyway this is a bit tricky...
>
>
> Yep, it’s the same with all “nested” solvers, fieldsplit, ASM, MG, you
> name it.
> You first need the initial PCSetUp() so that the bare minimum is put in
> place, then you have to fetch things yourself and adapt it to your needs.
> We had a similar discussion with the MEF++ people last week, there is
> currently no way around this, AFAIK.
>

Actually, I hated this as well, so I built a way around it _if_ you are
using a DM to define the problem. Then
you can set a "nullspace constructor" to make it if the field you are
talking about is ever extracted. You use DMSetNullSpaceConstructor(). I do
this in SNES ex62 and ex69, and other examples.

  Thanks,

 Matt


> Thanks,
> Pierre
>
> Regards,
> Nicolas
>
> --
> *De :* pierre.joli...@lip6.fr 
> *Envoyé :* mercredi 12 juillet 2023 19:52
> *À :* TARDIEU Nicolas 
> *Cc :* petsc-users@mcs.anl.gov 
> *Objet :* Re: [petsc-users] Near null space for a fieldsplit in petsc4py
>
>
> > On 12 Jul 2023, at 6:04 PM, TARDIEU Nicolas via petsc-users <
> petsc-users@mcs.anl.gov> wrote:
> >
> > Dear PETSc team,
> >
> > In the attached example, I set up a block pc for a saddle-point problem
> in petsc4py. The IS define the unknowns, namely some physical quantity
> (phys) and a Lagrange multiplier (lags).
> > I would like to attach a near null space to the physical block, in order
> to get the best performance from an AMG pc.
> > I have been trying hard, attaching it to the initial block, to the IS
> but no matter what I am doing, when it comes to "ksp_view", no near null
> space is attached to the matrix.
> >
> > Could you please help me figure out what I am doing wrong ?
>
> Are you using a double-precision 32-bit integers real build of PETSc?
> Is it --with-debugging=0?
> Because with my debug build, I get the following error (thus explaining
> why it’s not attached to the KSP).
> Traceback (most recent call last):
>   File "/Volumes/Data/Downloads/test/test_NullSpace.py", line 35, in
> 
> ns = NullSpace().create(True, [v], comm=comm)
>  
>   File "petsc4py/PETSc/Mat.pyx", line 5611, in
> petsc4py.PETSc.NullSpace.create
> petsc4py.PETSc.Error: error code 62
> [0] MatNullSpaceCreate() at
> /Volumes/Data/repositories/petsc/src/mat/interface/matnull.c:249
> [0] Invalid argument
> [0] Vector 0 must have 2-norm of 1.0, it is 22.3159
>
> Furthermore, if you set yourself the constant vector in the near
> null-space, then the first argument of create() must be False, otherwise,
> you’ll have twice the same vector, and you’ll end up with another error
> (the vectors in the near null-space must be orthonormal).
> If things still don’t work after those couple of fixes, please feel free
> to send an up-to-date reproducer.
>
> Thanks,
> Pierre
>
> > Thanks,
> > Nicolas
> >
> >
> >
> >
> > Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
> >
> > Si vous n'êtes pas le destinataire de ce Message, il vous est interdit
> de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
> >
> > Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> > 
> >
> > This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
> >
> > If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and notify the sender immediately by
> return message.
> >
> > E-m

Re: [petsc-users] (no subject)

[Matthew Knepley]

On Tue, Jul 11, 2023 at 3:58 PM Константин via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Hello, I'm trying to build petsc on windows. And when I make it I have
> such problem
>
>
Did you run configure first?

  Thanks,

 Matt


> --
> Константин
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Inquiry about the PetscScalar in Petsc.

[Matthew Knepley]

On Tue, Jul 11, 2023 at 11:48 AM neil liu  wrote:

> Thanks for your reply.
> I am using DMPLEX to read GMSH. And it seems I have to use
> DMGetCoordinatesLocal();
> VecGetArray(veccords, &coords);//it seems coords has to be defined by
> complex either.
>
> 2) Store the coordinates in coords with imaginary part equal to zero.
> I got the coordinates from above function *VecGetArray*, therefore will
> this reduce the memory?
>
> 3) is a good way to work around that.
>

We just store them with real part 0.

  Thanks,

Matt


> On Tue, Jul 11, 2023 at 11:40 AM Barry Smith  wrote:
>
>>
>>   Not really. PetscScalar (and entries of vectors etc) are either real or
>> complex, they cannot be mixed.
>>
>>   What parts of PETSc are you using, DM?
>>
>>   You can
>>
>> 1) store the coordinates in arrays and not use Vec for them. (of if you
>> are not using DM)
>> 2) sore them in Vec but with imaginary part 0
>> 3) rewrite your complex systems as real systems (with twice as many
>> unknowns, but only real ones)
>>
>>   Barry
>>
>>
>> > On Jul 11, 2023, at 11:34 AM, neil liu  wrote:
>> >
>> > Dear Petsc developers,
>> >
>> > My computation involves both real and complex numbers.
>> > For example, my coordinate is real, but my matrix is complex.
>> >
>> > I use VecGetArray to get the coordinates. But it seems coordinates have
>> to be defined in a complex manner to be consistent with complex matrix.
>> >
>> > Is there some ways to work around this if I understood correctly ?
>> >
>> > Thanks,
>> >
>> > Xiaodong
>> >
>> >
>> >
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ 

Re: [petsc-users] Solving a simpler PDE results in error while solving the original equation works in snes/tutorials/ex13.c

[Matthew Knepley]

On Mon, Jul 10, 2023 at 5:02 PM Tony Wu  wrote:

> In snes/tutorials/ex13.c,
> there is a function SetupPrimalProblem(), which sets up the f_0 and f_1 in
> PetscDSSetResidual() as described in
> https://petsc.org/release/manualpages/DT/PetscDSSetResidual/#petscdssetresidual
> ,
> and also the boundary conditions using the exact solution
> trig_inhomogeneous_u().
>
> Now, I am trying to modify f_0, f_1, and the boundary conditions in order
> to solve the 2D equation
> u+\Delta u=f,
> where f(x,y)=x and the boundary conditions are determined by the exact
> solution u(x,y)=x.
>
> In order to do this, I defined the following point-wise functions:
>
> static void f00(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt
> uOff[], const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar
> u_t[], const PetscScalar u_x[], const PetscInt aOff[], const PetscInt
> aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar
> a_x[], PetscReal t, const PetscReal x[], PetscInt numConstants, const
> PetscScalar constants[], PetscScalar f0[])
> {
>   f0[0] -= u[0];
>   f0[0] += x[0];
> }
>
> static void f11(PetscInt dim, PetscInt Nf, PetscInt NfAux, const PetscInt
> uOff[], const PetscInt uOff_x[], const PetscScalar u[], const PetscScalar
> u_t[], const PetscScalar u_x[], const PetscInt aOff[], const PetscInt
> aOff_x[], const PetscScalar a[], const PetscScalar a_t[], const PetscScalar
> a_x[], PetscReal t, const PetscReal x[], PetscInt numConstants, const
> PetscScalar constants[], PetscScalar f1[])
> {
>   PetscInt d;
>   for (d = 0; d < dim; ++d)
>   {
> f1[d] = u_x[d];
>   }
> }
>
> static void u_exact(PetscInt dim, PetscReal time, const PetscReal x[],
> PetscInt Nc, PetscScalar *u, void *ctx)
>

This function is declared "void" but should be "PetscErrorCode". There must
have been a warning on compile.

  Thanks,

 Matt


> {
>   *u = 0.0;
>   *u += x[0];
>   return PETSC_SUCCESS;
> }
>
> I then replaced the f0, f1_u and ex in SetupPrimalProblem()
> by the corresponding f00, f11, and u_exact.
> .
>
> However, I got the following error message:
>
> [0]PETSC ERROR: - Error Message
> --
> [0]PETSC ERROR:
> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.19.3, unknown
> [0]PETSC ERROR: ./ex13 on a arch-linux-c-debug named LAPTOP by user Tue
> Jul 11 04:06:13 2023
> [0]PETSC ERROR: Configure options --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3
> --FOPTFLAGS=-O3 --with-shared-libraries --with-cxx-dialect=C++11
> --with-make-np=8 --download-hdf5=yes --download-mumps=yes
> --download-parmetis=yes --download-metis=yes --download-superlu=yes
> --download-hypre=yes --download-superlu_dist=yes --download-scalapack=yes
> --download-triangle=yes
> [0]PETSC ERROR: #1 DMProjectPoint_Func_Private() at
> /home/user/petsc/src/dm/impls/plex/plexproject.c:128
> [0]PETSC ERROR: #2 DMProjectPoint_Private() at
> /home/user/petsc/src/dm/impls/plex/plexproject.c:395
> [0]PETSC ERROR: #3 DMProjectLocal_Generic_Plex() at
> /home/user/petsc/src/dm/impls/plex/plexproject.c:852
> [0]PETSC ERROR: #4 DMProjectFunctionLabelLocal_Plex() at
> /home/user/petsc/src/dm/impls/plex/plexproject.c:921
> [0]PETSC ERROR: #5 DMProjectFunctionLabelLocal() at
> /home/user/petsc/src/dm/interface/dm.c:8071
> [0]PETSC ERROR: #6 DMPlexInsertBoundaryValuesEssential() at
> /home/user/petsc/src/dm/impls/plex/plexfem.c:795
> [0]PETSC ERROR: #7 DMPlexInsertBoundaryValues_Plex() at
> /home/user/petsc/src/dm/impls/plex/plexfem.c:1046
> [0]PETSC ERROR: #8 DMPlexInsertBoundaryValues() at
> /home/user/petsc/src/dm/impls/plex/plexfem.c:1154
> [0]PETSC ERROR: #9 DMPlexSNESComputeBoundaryFEM() at
> /home/user/petsc/src/snes/utils/dmplexsnes.c:1372
> [0]PETSC ERROR: #10 SNESComputeFunction_DMLocal() at
> /home/user/petsc/src/snes/utils/dmlocalsnes.c:63
> [0]PETSC ERROR: #11 SNESComputeFunction() at
> /home/user/petsc/src/snes/interface/snes.c:2389
> [0]PETSC ERROR: #12 SNESSolve_NEWTONLS() at
> /home/user/petsc/src/snes/impls/ls/ls.c:172
> [0]PETSC ERROR: #13 SNESSolve() at
> /home/user/petsc/src/snes/interface/snes.c:4644
> [0]PETSC ERROR: #14 PetscConvEstGetConvRateSNES_Private() at
> /home/user/petsc/src/snes/utils/convest.c:369
> [0]PETSC ERROR: #15 PetscConvEstGetConvRate() at
> /home/user/petsc/src/snes/utils/convest.c:468
> [0]PETSC ERROR: #16 SNESSolve() at
> /home/user/petsc/src/snes/interface/snes.c:4561
> [0]PETSC ERROR: #17 main() at ex13.c:431
> [0]PETSC ERROR: Reached the main program with an out-of-range error code
> 584475600. This should never happen
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -convest_num_refine 2 (source: command line)
> [0]PETSC ERROR: -potential_petscspace_degree 1 (source: command line)
> [0]PETSC ERROR: -snes_convergence_estimate (source: command line)
> [0]PETSC ERROR: -snes_monitor (source: command line)
> [0]PETSC ERROR: End of Error M