On Thu, Jan 19, 2023 at 10:12 PM Nicholas Arnold-Medabalimi < narno...@umich.edu> wrote:
> Ok thanks for the clarification. In theory, if before the Reduction back > to the original distribution, if I call DMGlobaltoLocal then even with > MPI_REPLACE all the leafs corresponding to the original root should have > the same value so I won't have an ambiguity, correct? > That is right, so it should give you the result you expect. Thanks, Matt > On Thu, Jan 19, 2023 at 9:28 PM Matthew Knepley <knep...@gmail.com> wrote: > >> On Thu, Jan 19, 2023 at 9:13 PM Nicholas Arnold-Medabalimi < >> narno...@umich.edu> wrote: >> >>> Hi Matt >>> >>> Yep, that makes sense and is consistent. >>> >>> My question is a little more specific. So let's say I take an >>> initial mesh and distribute it and get the distribution SF with an overlap >>> of one. Consider a cell that is a root on process 0 and a leaf on process 1 >>> after the distribution. >>> >>> Will the distribution pointSF have an entry for the cell that is a leaf >>> in the ghost cell sense? >>> >>> I guess, in short does the distribution SF only have entries for the >>> movement of points that are roots in the ghost SF? >>> >> >> I do not understand the question. Suppose that a certain cell, say 0, in >> the original distribution goes to two different processes, say 0 and 1, and >> will happen when you distribute with overlap. Then the migration SF has two >> leaf entries for that cell, one from process 0 and one from process 1. They >> both point to root cell 0 on process 0. >> >> >>> Sorry if this is a little unclear. >>> >>> Maybe my usage will be a bit clearer. I am generating a distributionSF >>> (type 2 in your desc) then using that to generate a dof distribution(type >>> 3) using the section information. I then pass the information from the >>> initial distribution to new distribution with PetscSFBcast with >>> MPI_REPLACE. That scatters the vector to the new distribution. I then do >>> "stuff" and now want to redistribute back. So I pass the same dof >>> distributionSF but call PetscSFReduce with MPI_REPLACE. My concern is I am >>> only setting the root cell values on each partition. So if the ghost cells >>> are part of the distribution SF there will be multiple cells reducing to >>> the original distribution cell? >>> >> >> Yes, definitely. >> >> Thanks, >> >> Matt >> >> >>> Thanks >>> Nicholas >>> >>> >>> On Thu, Jan 19, 2023 at 8:28 PM Matthew Knepley <knep...@gmail.com> >>> wrote: >>> >>>> On Thu, Jan 19, 2023 at 11:58 AM Nicholas Arnold-Medabalimi < >>>> narno...@umich.edu> wrote: >>>> >>>>> Hi Petsc Users >>>>> >>>>> I'm working with a distribution start forest generated by >>>>> DMPlexDistribute and PetscSFBcast and Reduce to move data between the >>>>> initial distribution and the distribution generated by DMPlex Distribute. >>>>> >>>>> I'm trying to debug some values that aren't being copied properly and >>>>> wanted to verify I understand how a redistribution SF works compared with >>>>> a >>>>> SF that describes overlapped points. >>>>> >>>>> [0] 0 <- (0,7) point 0 on the distributed plex is point 7 on >>>>> process 0 on the initial distribution >>>>> [0] 1 <- (0,8) point 1 on the distributed plex is point 8 on >>>>> process 0 on the initial distribution >>>>> [0] 2 <- (0,9) >>>>> [0] 3 <- (0,10) >>>>> [0] 4 <- (0,11) >>>>> >>>>> [1] 0 <- (1,0) point 0 on the distributed plex is point 0 on >>>>> process 1 on the initial distribution >>>>> [1] 1 <- (1,1) >>>>> [1] 2 <- (1,2) >>>>> [1] 3 <- (0,0) point 3 on the distributed plex is point 0 on >>>>> process 0 on the initial distribution >>>>> [1] 4 <- (0,1) >>>>> [1] 5 <- (0,2) >>>>> >>>>> my confusion I think is how does the distributionSF inform of what >>>>> cells will be leafs on the distribution? >>>>> >>>> >>>> I should eventually write something to clarify this. I am using SF in >>>> (at least) two different ways. >>>> >>>> First, there is a familiar SF that we use for dealing with "ghost" >>>> points. These are replicated points where one process >>>> is said to "own" the point and another process is said to hold a >>>> "ghost". The ghost points are leaves in the SF which >>>> point back to the root point owned by another process. We call this the >>>> pointSF for a DM. >>>> >>>> Second, we have a migration SF. Here the root points give the original >>>> point distribution. The leaf points give the new >>>> point distribution. Thus a PetscSFBcast() pushes points from the >>>> original to new distribution, which is what we mean >>>> by a migration. >>>> >>>> Third, instead of point values, we might want to communicate fields >>>> over those points. For this we make new SFes, >>>> where the numbering does not refer to points, but rather to dofs. >>>> >>>> Does this make sense? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Sincerely >>>>> Nicholas >>>>> >>>>> -- >>>>> Nicholas Arnold-Medabalimi >>>>> >>>>> Ph.D. Candidate >>>>> Computational Aeroscience Lab >>>>> University of Michigan >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >>> >>> -- >>> Nicholas Arnold-Medabalimi >>> >>> Ph.D. Candidate >>> Computational Aeroscience Lab >>> University of Michigan >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > > > -- > Nicholas Arnold-Medabalimi > > Ph.D. Candidate > Computational Aeroscience Lab > University of Michigan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>