On Sun, Jul 3, 2022 at 3:34 PM Nicholas Arnold-Medabalimi < narno...@umich.edu> wrote:
> Hi > > Thanks for the response. I still have a bit of confusion here. To make > sure I follow I have setup a code block (at end) with options for overlap > and CreateGhostCells. > > Using a 2x2x2 Box mesh I've seen the following behavior. > > When I use overlap=1 I get approximately what I would expect. > If I run on 1 processor there are only 8cells and everything is normal > If I run on 2 processors each processor has 8cells (4 from the split > domain and an extra 4 extending into the neighboring cell) > My question here is how do I then differentiate the overlap cells from > each processor so that I can synchronize the extra cells as I iterate? > Both GetHeightStratum GetSimplex will identify the 8cells with no > differentiation between the partitions vs the overlap. > I believe this is the strategy that will get me the result I want where I > can synch the overlaps so that when I iterate through the "owned cells" the > stencil can reach into the overlap cells to get the information they need? > You can tell whether a cell is in the overlap by seeing whether it is in the point SF. A PetscSF (Star Forest) is an object that relates mesh points on two different processes. It is a directed arrow that we see as pointing from a leaf (ghost cell) to a root (owned cell). So you can say DMPlexGetHeightStratum(plex, 0, &cStart, &cEnd); DMPlexGetPointSF(plex, &sf); PetscSFGetGraph(sf, NULL, &Nl, &leaves, NULL); for (c = cStart; c < cEnd; ++c) { PetscFindInt(c, Nl, leaves, &loc); if (loc >= 0) /* cell is in the overlap */ } > When I use ghostCellGenerate I also get what I would expect > If I run on 1 processor each of the outside faces of the cube gets an > extra cell attached so I get Box Cells 0 through 8 and then Ghost Cells 8 > through 32. > If I run on 2 processors only the outside faces of the cub get the extra > cells so I get Box Cells 0 through 4 and then Ghost Cells 4 through 16 > So I don't think this is what I want for my intended use? > I do not understand the last sentence. Thanks, Matt > Thanks Again > > Sincerely > Nicholas > > PetscInt facecount = 10; > PetscInt i, dim = 3, overlap = 0; > ierr = PetscOptionsGetInt(NULL, NULL, "-facecount", &facecount, NULL); > CHKERRQ(ierr); > for (i = 0; i < dim; i++) > faces[i] = facecount; > PetscViewerASCIIPrintf(viewerstdout, "facecount is %d\n", facecount); > PetscReal lower[3] = {0.0, 0.0, 0.0}; > PetscReal upper[3] = {2.0, 2.0, 2.0}; > ierr = DMPlexCreateBoxMesh(comm, dim, simplex, faces, lower, upper, > /* periodicity */ NULL, dmInterped, &dm); > CHKERRQ(ierr); > ierr = DMPlexDistribute(dm, overlap, &distributionSF, &dmDist); > CHKERRQ(ierr); > if (dmDist) > { > ierr = DMDestroy(&dm); > CHKERRQ(ierr); > dm = dmDist; > } > > PetscInt ghostCells; > > PetscBool use_ghostcells = PETSC_TRUE; > ierr = PetscOptionsGetBool(NULL, NULL, "-use_ghostcells", &use_ghostcells, > NULL); > DM dmGhost; > if (use_ghostcells) > { > ierr = DMPlexConstructGhostCells(dm, NULL, &ghostCells, &dmGhost); > CHKERRQ(ierr); > if (dmGhost) > { > PetscViewerASCIIPrintf(viewerstdout, "%dghost Cells generated > \n", ghostCells); > ierr = DMDestroy(&dm); > CHKERRQ(ierr); > dm = dmGhost; > } > else > PetscViewerASCIIPrintf(viewerstdout, "ghost Cells not > generated\n"); > } > else > PetscViewerASCIIPrintf(viewerstdout, "ghost Cells not generated\n" > ); > > PetscInt c0, c1, f0, f1, e0, e1, v0, v1, s0, s1, g0, g1; > > DMPlexGetHeightStratum(dm, 0, &c0, &c1); > DMPlexGetHeightStratum(dm, 1, &f0, &f1); > DMPlexGetHeightStratum(dm, 2, &e0, &e1); > DMPlexGetHeightStratum(dm, 3, &v0, &v1); > DMPlexGetSimplexOrBoxCells(dm, 0, &s0, &s1); > DMPlexGetGhostCellStratum(dm, &g0, &g1); > PetscViewerASCIIPrintf(viewerstdout, "Cells: p0:%d ,p1:%d \n", c0, > c1); > PetscViewerASCIIPrintf(viewerstdout, "Faces: p0:%d ,p1:%d \n", f0, > f1); > PetscViewerASCIIPrintf(viewerstdout, "Edges: p0:%d ,p1:%d \n", e0, > e1); > PetscViewerASCIIPrintf(viewerstdout, "Vertices: p0:%d ,p1:%d \n", v0, > v1); > PetscViewerASCIIPrintf(viewerstdout, "Box Cells: p0:%d ,p1:%d \n", > s0, s1); > PetscViewerASCIIPrintf(viewerstdout, "Ghost Cells: p0:%d ,p1:%d \n", > g0, g1); > > > > On Sun, Jul 3, 2022 at 12:42 AM Matthew Knepley <knep...@gmail.com> wrote: > >> On Sat, Jul 2, 2022 at 6:20 PM Nicholas Arnold-Medabalimi < >> narno...@umich.edu> wrote: >> >>> Dear PETSc users, >>> >>> I have some general novice questions regarding how to properly access a >>> distributed DMPLEX vector that has ghost cells for a finite volume use case. >>> >>> My process for setup >>> 1) generated a simple box mesh using DMPlexCreateBoxMesh >>> 2) distribute the mesh using DMPlexDistribute (at this point everything >>> looks fine when I View (as paraview file) and I can see the partitions) >>> 3) I generate a section with 5 variables located at the cells. (I don't >>> think it matters if I use 5 fields or 1 field with 5 components?) I can >>> alternatively do this using the PetscFV variable with 5 components. >>> 4) I then have the dm setup properly (with no ghost cells at this >>> point) and I use DMCreateGlobalVector and DMCreateLocalVector to get the >>> Vectors to work with >>> 5) I'm just setting a quadratic IC. The overall loop over the cells is >>> constructed via getting the cell stratum which has the Cell point list and >>> then using the SectionOffset to assign each of the components at each cell >>> using DMPlexComputeCellGeometryFVM to get the cell centroid. >>> >>> As far as I can tell this is an adequate solution but now I'm moving >>> into learning how to access neighboring cells for flux calculations which >>> isn't just the local cell but requires access to neighboring cells which at >>> some point will be located on neighboring processors and synched to be >>> accessible. When I did this for a DMDA it was pretty straightforward >>> insofar as I just needed to call DMDAGetCorners and the LocalVector >>> indexing would allow the stencil to extend gracefully into the ghost >>> values. >>> >>> My general question is how do I achieve a similar loop structure for >>> DMPlex as far as the indexing over the owned cells while being able to >>> access the neighbor halo cells. I have been looking at example 52 and 11 in >>> the ts tutorial examples but I'm struggling to extract exactly what I need. >>> >>> To boil it down into 3 starting questions >>> 1) What is the preferred way to setup the cells and then access the >>> internal cells for say N cell halos for the boundaries between processors >>> in a DMPlex? >>> >> >> If you want a halo during distribution, you can give the "overlap" >> argument to DMPlexDistribute(). >> >> >>> 2) What is the distinction between the overlap argument in >>> DMPlexDistribute and DMPlexConstructGhostCells (which I see used in the >>> examples)? (at least based on my tests it seems GhostCells is generating >>> cells at the overall boundaries not the internal partition boundaries?) >>> >> >> The examples use both. DMPlexConstructGhostCells() puts a layer of cells >> around the domain boundary that are special. They have no cell boundary, >> just the face they are attached to. The overlap adds true cells to each >> domain across the parallel boundary. >> >> >>> 3) Once I do have a DMPlex object that has the appropriate halos, how >>> would I get the access range for those cells? Based on my current >>> assumption the Stratum range for the Cells would include the ghost cells as >>> well which I would like to avoid? >>> >> >> I use >> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetSimplexOrBoxCells >> to avoid the FV ghost cells. However, you can also do it by hand by >> checking the type, or use >> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGhostCellStratum >> >> Thanks, >> >> Matt >> >> >>> (Ultimately in the examples I see cases where there is the synch step >>> for Local to Global and vice versa but I'm trying to just get to the point >>> where I have the DM setup properly to handle the internal mesh boundaries >>> and have the correct indexing from the StratumGet commands.) >>> >>> Any advice and corrections would be welcome. >>> >>> Sincerely >>> Nicholas >>> >>> -- >>> Nicholas Arnold-Medabalimi >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > > > -- > Nicholas Arnold-Medabalimi > > Ph.D. Candidate > Computational Aeroscience Lab > University of Michigan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>