On Fri, Jul 15, 2022 at 7:05 PM Bora Jeong <boraj1...@gmail.com> wrote:
> I found that iroots() and ileaves() have size of nleaves and tested > through the attached sample code with the grid previously shared. Then I > get the results written in the attached monitor file. > > It seems ileaves(i)%rank and ileaves(i)%index at line 127 of the attached > code have garbage values, different from displayed values by PetscSFView() > at line 113. > > It is tough to get it why the garbage values are returned from > PetscSFGetGraph(). Any comments will be very appreciated. > Unfortunately, Fortran is not very good at reporting declaration errors. The problem is that you did not include or use the Vec module. I have done this and your example runs for me. I have included the modified code. Thanks, Matt > Best > > On Fri, Jul 15, 2022 at 3:53 PM Matthew Knepley <knep...@gmail.com> wrote: > >> On Fri, Jul 15, 2022 at 2:25 PM Bora Jeong <boraj1...@gmail.com> wrote: >> >>> Thank you for the comments. Updated sample code is attached here; >>> DMPlexDistribute() is explicitly called and by defining a section before >>> mesh distribution, a Natural SF was able to be defined as found from >>> PetscSFView(). >>> >>> However, I am still struggling to call PetscSFGetGraph() >>> https://petsc.org/main/docs/manualpages/PetscSF/PetscSFGetGraph/ >>> due to data type definition of ilocal and iremote. What is the proper >>> size allocation for those two variables? Does this size for the number of >>> processors? or the number of vertex? >>> >> >> Since we need to pass back arrays, you need to pass us in F90 pointers. >> Here is an example of doing such a thing: >> >> >> https://gitlab.com/petsc/petsc/-/blob/main/src/vec/is/sf/tutorials/ex1f.F90#L94 >> >> Thanks, >> >> Matt >> >> >>> Best >>> >>> On Fri, Jul 15, 2022 at 9:09 AM Matthew Knepley <knep...@gmail.com> >>> wrote: >>> >>>> On Fri, Jul 15, 2022 at 8:46 AM Bora Jeong <boraj1...@gmail.com> wrote: >>>> >>>>> Okay, I got it what's happening. First, this very bright functionality >>>>> of petsc (the natural-global converting) needs to be documented in a >>>>> better >>>>> way for sure. Currently, it is very difficult to use/follow this features >>>>> as an external user. Hope this will move forward in a better way. >>>>> >>>>> Then next question is if I load/distribute mesh just like I am doing >>>>> right now shown in the sample code, can I assume that my code does not >>>>> create natural sf during the distribution(always)? In other words, can I >>>>> always get the natural order of each node by simply stacking the preceding >>>>> processor's number of node? For example, for proc=0, natural node ID might >>>>> be just 1 to nnode_proc_0, >>>>> for proc=1, it might be {nnode_proc_0 + 1 to nnode_proc_0 + >>>>> nnode_proc_1} and so on. >>>>> >>>>> Does that always make sense in this case? >>>>> >>>> >>>> No, but if you call DMPlexDistribute() yourself, rather than having it >>>> called automatically by DMSetFromOptions(), you can >>>> preserve the mapping: >>>> >>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexDistribute/ >>>> >>>> The SF returned maps the original point distribution, which is in the >>>> same order as the file, to the redistributed points, which are determined >>>> by the mesh partitioner. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Best >>>>> >>>>> >>>>> On Fri, Jul 15, 2022 at 8:07 AM Matthew Knepley <knep...@gmail.com> >>>>> wrote: >>>>> >>>>>> On Fri, Jul 15, 2022 at 7:17 AM Bora Jeong <boraj1...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> A sample code for loading dm and declaring Natural SF from a sphere >>>>>>> mesh is attached here. PetscSFView() returns NULL from sf_nat. >>>>>>> >>>>>> >>>>>> The Global-to-Natural mapping relates global dofs to natural dofs. >>>>>> Thus, in order to compute this mapping the DM has to know >>>>>> about the dof layout before distribution. This means you need to >>>>>> setup a local section before distributing, as we do for example in >>>>>> Plex test ex15. This makes things more complicated since everything >>>>>> cannot be packaged up into DMSetFromOptions(), but we need >>>>>> user input so I do not see a simpler way to do things. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Best >>>>>>> >>>>>>> On Fri, Jul 15, 2022 at 6:39 AM Matthew Knepley <knep...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Jul 14, 2022 at 8:25 PM Bora Jeong <boraj1...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Okay, I checked it and you are correct. In my case, simply, >>>>>>>>> natural node index can be identified by stacking all the preceding >>>>>>>>> processor's numbers of nodes for a particular processor, which is >>>>>>>>> good due >>>>>>>>> to simplicity. However, one serious question is why this is happening >>>>>>>>> in my >>>>>>>>> code? In other words, why the natural SF is not created during the >>>>>>>>> mesh >>>>>>>>> distribution? My code wants to have consistency in dealing with this >>>>>>>>> natural indexing for several different kinds of mesh files. So there >>>>>>>>> is a >>>>>>>>> necessity to guarantee of consistency in this weird behavior. >>>>>>>>> >>>>>>>> >>>>>>>> I can't tell what is going on in your code unless I can run it. Do >>>>>>>> you have a simple example? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> On Thu, Jul 14, 2022 at 6:43 PM Matthew Knepley <knep...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Jul 14, 2022 at 5:47 PM Bora Jeong <boraj1...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thank you for the comments. I have these errors when I call >>>>>>>>>>> PetscSFView() after DMGetNaturalSF() >>>>>>>>>>> >>>>>>>>>>> [2]PETSC ERROR: --------------------- Error Message >>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>> [2]PETSC ERROR: Null argument, when expecting valid pointer >>>>>>>>>>> [2]PETSC ERROR: Null Pointer: Parameter # 1 >>>>>>>>>>> [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>>>>>>>>> shooting. >>>>>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.17.0, unknown >>>>>>>>>>> [2]PETSC ERROR: #1 PetscSFView() at >>>>>>>>>>> [2]PETSC ERROR: #2 User provided function() at User file:0 >>>>>>>>>>> Abort(85) on node 2 (rank 0 in comm 16): application called >>>>>>>>>>> MPI_Abort(MPI_COMM_SELF, 85) - process 0 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Clearly NULL was returned, which means no natural SF was created. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Below is the structure to load a mesh file from gmsh; >>>>>>>>>>> >>>>>>>>>>> call DMCreate(PETSC_COMM_WORLD, dm, ierr);CHKERRA(ierr) >>>>>>>>>>> call DMSetType(dm, plex, ierr);CHKERRA(ierr) >>>>>>>>>>> call DMSetUseNatural(dm, PETSC_TRUE, ierr);CHKERRA(ierr) >>>>>>>>>>> call DMSetFromOptions(dm, ierr);CHKERRA(ierr) >>>>>>>>>>> call DMGetNaturalSF(dm, sf_nat, ierr);CHKERRA(ierr) >>>>>>>>>>> call PetscSFView(sf_nat, PETSC_VIEWER_STDOUT_WORLD, >>>>>>>>>>> ierr);CHKERRA(ierr) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The natural SF is created during mesh distribution. That has not >>>>>>>>>> happened here. This means that >>>>>>>>>> the order of cells is identical to the file it was read from. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Best >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 14, 2022 at 10:49 AM Matthew Knepley < >>>>>>>>>>> knep...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Wed, Jul 13, 2022 at 10:17 PM Bora Jeong < >>>>>>>>>>>> boraj1...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Dear petsc team, >>>>>>>>>>>>> >>>>>>>>>>>>> I am a user of DMPlex for a finite volume code and there is a >>>>>>>>>>>>> necessity to know global index of each cell. Here the global >>>>>>>>>>>>> index means >>>>>>>>>>>>> the indexing that can be found from a mesh file itself without >>>>>>>>>>>>> distribution >>>>>>>>>>>>> over processors. It seems petsc community denotes this indexing >>>>>>>>>>>>> term as >>>>>>>>>>>>> "natural". >>>>>>>>>>>>> >>>>>>>>>>>>> What I want to do is to create a local array (not petsc vector >>>>>>>>>>>>> but just an array variable in the program) to map distributed >>>>>>>>>>>>> cell ID to >>>>>>>>>>>>> natual cell ID, for example, an array "A"; >>>>>>>>>>>>> A(distributed_node_ID) = natural_node_ID >>>>>>>>>>>>> >>>>>>>>>>>>> There are some petsc functions to support mapping between >>>>>>>>>>>>> global and natural vectors. However, I just need to define the >>>>>>>>>>>>> array "A" as >>>>>>>>>>>>> above example. To achieve this, what is a proper/smart way? In >>>>>>>>>>>>> other words, >>>>>>>>>>>>> how can I extract the natural_cell_ID from a distributed >>>>>>>>>>>>> local_cell_ID? >>>>>>>>>>>>> >>>>>>>>>>>>> I turned on DMSetUseNatural(DM, PETSC_TRUE) before >>>>>>>>>>>>> distribution, but after this, defining all the required section >>>>>>>>>>>>> and star >>>>>>>>>>>>> forest objects to get natural and global vectors seems not that >>>>>>>>>>>>> direct way >>>>>>>>>>>>> for my purpose, which is just to extract the above mapping array >>>>>>>>>>>>> "A". Can I >>>>>>>>>>>>> get any comments about it? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> There is only one thing created, the sfNatural PetscSF object, >>>>>>>>>>>> which you can get with DMGetNaturalSF(). The roots of this SF are >>>>>>>>>>>> the global numbers of dofs stored in PETSc vectors, and the >>>>>>>>>>>> leaves are natural numbers for these dofs. Thus, when we map global >>>>>>>>>>>> vectors to natural vectors in DMPlexGlobalToNaturalBegin/End(), >>>>>>>>>>>> we call PetscSFBcastBegin/End(). Mapping natural to global we call >>>>>>>>>>>> PetscSFReduceBegin/End(). You could pull the information out of >>>>>>>>>>>> the SF using PetscSFGetGraph() if you want. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Regards >>>>>>>>>>>>> Mo >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>> their experiments is infinitely more interesting than any results >>>>>>>>>>>> to which >>>>>>>>>>>> their experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>>> their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
test.F90
Description: Binary data