I am doing that as well (although not for vertex dofs). And I had it working quite well for purely cell-associated DOFs. But I realized later that I also wanted to transmit some DOFs associated with faces so I suspect I'm messing something up there.
Something we discussed back on 12/26 (email subject:Getting a vector from a DM to output VTK) was associating a vector generated this way with the DM so it can be more easily visualized using vtk files is VecSetOperation(state_dist, VECOP_VIEW, (void(*)(void))VecView_Plex); If there is an easy way to get this working in Fortran I would very much appreciate it as it was very helpful when I was debugging in C. Thanks Nicholas On Fri, Jan 6, 2023 at 11:37 AM Matthew Knepley <knep...@gmail.com> wrote: > On Fri, Jan 6, 2023 at 11:32 AM Nicholas Arnold-Medabalimi < > narno...@umich.edu> wrote: > >> Hi Matt >> >> This was generated using the DMPlexDistributeField which we discussed a >> while back. Everything seemed to be working fine when I only had cells dofs >> but I recently added face dofs, which seems to have caused some issues. >> Whats weird is that I'm feeding the same distribution SF and inputting a >> vector and section that are consistent to the DMPlexDistributeField so I'd >> expect the vectors and section output to be consistent. I'll take a closer >> look at that. >> > > The way I use DistributeField(), for example to distribute coordinates, is > that I give it the migrationSF, the local coordinate section, and the local > coordinate vector. It gives me back the new local coordinate section (which > I set into the distributed DM), and the local coordinate vector. It seems > like you are doing something else. Maybe the documentation is misleading. > > Thanks, > > Matt > > >> Thanks >> Nicholas >> >> On Fri, Jan 6, 2023 at 10:59 AM Matthew Knepley <knep...@gmail.com> >> wrote: >> >>> On Fri, Jan 6, 2023 at 10:41 AM Nicholas Arnold-Medabalimi < >>> narno...@umich.edu> wrote: >>> >>>> Hi Matt >>>> >>>> I appreciate the help. The section view is quite extensive because each >>>> cell has 55 dofs located at the cells and on certain faces. I've appended >>>> the first of these which corresponds with the output in the first email, to >>>> save space. The following 54 are exactly the same but offset incremented by >>>> 1. (or negative 1 for negative offsets) >>>> >>> >>> Okay, from the output it is clear that this vector does not match your >>> global section. Did you get stateVec by calling DMCreateGlobalVector()? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks for your time >>>> Nicholas >>>> >>>> On Fri, Jan 6, 2023 at 10:23 AM Matthew Knepley <knep...@gmail.com> >>>> wrote: >>>> >>>>> On Fri, Jan 6, 2023 at 10:10 AM Nicholas Arnold-Medabalimi < >>>>> narno...@umich.edu> wrote: >>>>> >>>>>> Hi Matt >>>>>> >>>>>> I apologize for any lack of clarity in the initial email. >>>>>> >>>>>> looking at the initial output on rank 1 >>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset-oStart >>>>>> cell 0 offset 2475 oStart 2640 -165 >>>>>> cell 1 offset 2530 oStart 2640 -110 >>>>>> cell 2 offset 2585 oStart 2640 -55 >>>>>> cell 3 offset 2640 oStart 2640 0 >>>>>> ..... >>>>>> cell 15 offset -771 oStart 2640 -3411 >>>>>> >>>>>> >>>>>> cell 15 provides a negative offset because it is the overlap cell >>>>>> (that is unowned) >>>>>> The remained of cells are all owned. However, the first 3 cells >>>>>> (0,1,2) return an offset that is less than the starting ownership range. >>>>>> I >>>>>> would expect cell 0 to start at offset 2640 at minimum. >>>>>> >>>>> >>>>> Send the output for this section >>>>> >>>>> call PetscSectionView(section, PETSC_VIEWER_STDOUT_WORLD); >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Sincerely >>>>>> Nicholas >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 6, 2023 at 10:05 AM Matthew Knepley <knep...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> On Fri, Jan 6, 2023 at 9:56 AM Nicholas Arnold-Medabalimi < >>>>>>> narno...@umich.edu> wrote: >>>>>>> >>>>>>>> Apologies. If it helps, there is one cell of overlap in this small >>>>>>>> test case for a 2D mesh that is 1 cell in height and a number of cells >>>>>>>> in >>>>>>>> length. . >>>>>>>> >>>>>>>> process 0 >>>>>>>> Petsc VecGetLocalSize 2750 >>>>>>>> size(stateVecV) 2750 >>>>>>>> >>>>>>>> process 1 >>>>>>>> Petsc VecGetLocalSize 2640 >>>>>>>> size(stateVecV) 2640 >>>>>>>> >>>>>>> >>>>>>> The offsets shown below are well-within these sizes. I do not >>>>>>> understand the problem. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> On Fri, Jan 6, 2023 at 9:51 AM Matthew Knepley <knep...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Fri, Jan 6, 2023 at 9:37 AM Nicholas Arnold-Medabalimi < >>>>>>>>> narno...@umich.edu> wrote: >>>>>>>>> >>>>>>>>>> Hi Matt >>>>>>>>>> >>>>>>>>>> I made a typo on the line statVecV(offset) = <set to something> >>>>>>>>>> in my example, I agree. (I wrote that offhand since the actual >>>>>>>>>> assignment >>>>>>>>>> is much larger) I should be statVecV(offset+1) = <assignment> so I'm >>>>>>>>>> confident it's not a 1 0 indexing thing. >>>>>>>>>> >>>>>>>>>> My question is more related to what is happening in the offsets. >>>>>>>>>> c0 and c1 are pulled using DMplexgetheight stratum, so they are >>>>>>>>>> zero-indexed (which is why I loop from c0 to (c1-1)). >>>>>>>>>> >>>>>>>>>> For the size inquiries. on processor 0 >>>>>>>>>> Petsc VecGetSize(stateVec) 5390 >>>>>>>>>> >>>>>>>>> >>>>>>>>> I need to see VecGetLocalSize() >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> size(stateVecV) 2640 >>>>>>>>>> >>>>>>>>>> on processor 1 >>>>>>>>>> Petsc VecGetSize 5390 >>>>>>>>>> size(stateVecV) 2750 >>>>>>>>>> >>>>>>>>>> It's quite weird to me that processor one can have a positive >>>>>>>>>> offset that is less than its starting ownership index (in the >>>>>>>>>> initial email >>>>>>>>>> output). >>>>>>>>>> >>>>>>>>>> Thanks for the assistance >>>>>>>>>> Nicholas >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jan 6, 2023 at 9:20 AM Matthew Knepley <knep...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Fri, Jan 6, 2023 at 2:28 AM Nicholas Arnold-Medabalimi < >>>>>>>>>>> narno...@umich.edu> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Petsc Users, >>>>>>>>>>>> >>>>>>>>>>>> I'm working with a dmplex system with a subsampled mesh >>>>>>>>>>>> distributed with an overlap of 1. >>>>>>>>>>>> >>>>>>>>>>>> I'm encountering unusual situations when using >>>>>>>>>>>> VecGetOwnershipRange to adjust the offset received from a global >>>>>>>>>>>> section. >>>>>>>>>>>> The logic of the following code is first to get the offset needed >>>>>>>>>>>> to index >>>>>>>>>>>> a global vector while still being able to check if it is an >>>>>>>>>>>> overlapped cell >>>>>>>>>>>> and skip if needed while counting the owned cells. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> call DMGetGlobalSection(dmplex,section,ierr) >>>>>>>>>>>> call VecGetArrayF90(stateVec,stateVecV,ierr) >>>>>>>>>>>> call VecGetOwnershipRange(stateVec,oStart,oEnd,ierr) >>>>>>>>>>>> do i = c0, (c1-1) >>>>>>>>>>>> >>>>>>>>>>>> call PetscSectionGetOffset(section,i,offset,ierr) >>>>>>>>>>>> write(*,*) "cell",i,"offset",offset,'oStart',oStart, offset >>>>>>>>>>>> -oStart >>>>>>>>>>>> >>>>>>>>>>>> if(offset<0) then >>>>>>>>>>>> cycle >>>>>>>>>>>> endif >>>>>>>>>>>> offset=offset-oStart >>>>>>>>>>>> plexcells=plexcells+1 >>>>>>>>>>>> stateVecV(offset)= <set to something> enddo >>>>>>>>>>>> >>>>>>>>>>>> I'm noticing some very weird results that I've appended below. >>>>>>>>>>>> The GetOffset documentation notes that a negative offset indicates >>>>>>>>>>>> an >>>>>>>>>>>> unowned point (which I use to cycle). However, the offset >>>>>>>>>>>> subtraction with >>>>>>>>>>>> oStart will yield an illegal index for the Vector access. I see >>>>>>>>>>>> that on the >>>>>>>>>>>> documentation for GetOwnershipRange, it notes that this may be >>>>>>>>>>>> "ill-defined" but I wanted to see if this is type of ill-defined >>>>>>>>>>>> I can >>>>>>>>>>>> expect or there is just something terribly wrong with my >>>>>>>>>>>> PetscSection.(both >>>>>>>>>>>> the Vec and Section were produced from DMPlexDistributeField so >>>>>>>>>>>> should by >>>>>>>>>>>> definition have synchronized section information) I was wondering >>>>>>>>>>>> if there >>>>>>>>>>>> is a possible output and/or the best way to index the vector. I'm >>>>>>>>>>>> thinking >>>>>>>>>>>> of subtracting the offset of cell 0 perhaps? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Can you show your vector sizes? Are you sure it is not the fact >>>>>>>>>>> that F90 arrays use 1-based indices, but these are 0-based offsets? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> on rank 0 >>>>>>>>>>>> >>>>>>>>>>>> cell 0 offset 0 oStart 0 >>>>>>>>>>>> 0 >>>>>>>>>>>> cell 1 offset 55 oStart 0 >>>>>>>>>>>> 55 >>>>>>>>>>>> cell 2 offset 110 oStart 0 >>>>>>>>>>>> 110 >>>>>>>>>>>> cell 3 offset 165 oStart 0 >>>>>>>>>>>> 165 >>>>>>>>>>>> cell 4 offset 220 oStart 0 >>>>>>>>>>>> 220 >>>>>>>>>>>> cell 5 offset 275 oStart 0 >>>>>>>>>>>> 275 >>>>>>>>>>>> cell 6 offset 330 oStart 0 >>>>>>>>>>>> 330 >>>>>>>>>>>> cell 7 offset 385 oStart 0 >>>>>>>>>>>> 385 >>>>>>>>>>>> cell 8 offset 440 oStart 0 >>>>>>>>>>>> 440 >>>>>>>>>>>> cell 9 offset 495 oStart 0 >>>>>>>>>>>> 495 >>>>>>>>>>>> cell 10 offset 550 oStart 0 >>>>>>>>>>>> 550 >>>>>>>>>>>> cell 11 offset 605 oStart 0 >>>>>>>>>>>> 605 >>>>>>>>>>>> cell 12 offset 660 oStart 0 >>>>>>>>>>>> 660 >>>>>>>>>>>> cell 13 offset 715 oStart 0 >>>>>>>>>>>> 715 >>>>>>>>>>>> >>>>>>>>>>>> and on rank one >>>>>>>>>>>> cell 0 offset 2475 oStart 2640 >>>>>>>>>>>> -165 >>>>>>>>>>>> cell 1 offset 2530 oStart 2640 >>>>>>>>>>>> -110 >>>>>>>>>>>> cell 2 offset 2585 oStart 2640 >>>>>>>>>>>> -55 >>>>>>>>>>>> cell 3 offset 2640 oStart 2640 >>>>>>>>>>>> 0 >>>>>>>>>>>> cell 4 offset 2695 oStart 2640 >>>>>>>>>>>> 55 >>>>>>>>>>>> cell 5 offset 2750 oStart 2640 >>>>>>>>>>>> 110 >>>>>>>>>>>> cell 6 offset 2805 oStart 2640 >>>>>>>>>>>> 165 >>>>>>>>>>>> cell 7 offset 2860 oStart 2640 >>>>>>>>>>>> 220 >>>>>>>>>>>> cell 8 offset 2915 oStart 2640 >>>>>>>>>>>> 275 >>>>>>>>>>>> cell 9 offset 2970 oStart 2640 >>>>>>>>>>>> 330 >>>>>>>>>>>> cell 10 offset 3025 oStart 2640 >>>>>>>>>>>> 385 >>>>>>>>>>>> cell 11 offset 3080 oStart 2640 >>>>>>>>>>>> 440 >>>>>>>>>>>> cell 12 offset 3135 oStart 2640 >>>>>>>>>>>> 495 >>>>>>>>>>>> cell 13 offset 3190 oStart 2640 >>>>>>>>>>>> 550 >>>>>>>>>>>> cell 14 offset 3245 oStart 2640 >>>>>>>>>>>> 605 >>>>>>>>>>>> cell 15 offset -771 oStart 2640 >>>>>>>>>>>> -3411 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Sincerely >>>>>>>>>>>> Nicholas >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Nicholas Arnold-Medabalimi >>>>>>>>>>>> >>>>>>>>>>>> Ph.D. Candidate >>>>>>>>>>>> Computational Aeroscience Lab >>>>>>>>>>>> University of Michigan >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>> which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Nicholas Arnold-Medabalimi >>>>>>>>>> >>>>>>>>>> Ph.D. Candidate >>>>>>>>>> Computational Aeroscience Lab >>>>>>>>>> University of Michigan >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Nicholas Arnold-Medabalimi >>>>>>>> >>>>>>>> Ph.D. Candidate >>>>>>>> Computational Aeroscience Lab >>>>>>>> University of Michigan >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nicholas Arnold-Medabalimi >>>>>> >>>>>> Ph.D. Candidate >>>>>> Computational Aeroscience Lab >>>>>> University of Michigan >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >>>> >>>> -- >>>> Nicholas Arnold-Medabalimi >>>> >>>> Ph.D. Candidate >>>> Computational Aeroscience Lab >>>> University of Michigan >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> >> >> -- >> Nicholas Arnold-Medabalimi >> >> Ph.D. Candidate >> Computational Aeroscience Lab >> University of Michigan >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > -- Nicholas Arnold-Medabalimi Ph.D. Candidate Computational Aeroscience Lab University of Michigan
