Take a look at these posts from last year and see if they will help you at least get a slice:
https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043037.html <https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043037.html> https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043043.html <https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043043.html> Good luck, Randy M. > On Jan 25, 2022, at 2:52 PM, Bastian Löhrer <bastian.loeh...@tu-dresden.de> > wrote: > > Dear PETSc community, > > in our CFD code we use a 3D DMDA to organize our data. > > Now I need to compute a derived quantity in a subregion of the global domain > and to write these data to disk for further post-processing. > > The subregion is actually a planar slice for now, but it could also be a > boxed-shaped region in future applications. > > Hence, I figured I would create a new DMDA for this subregion by writing > something along the lines of > > call DMDACreate3d( & ! > https://www.mcs.anl.gov/petsc/petsc-3.8.4/docs/manualpages/DMDA/DMDACreate3d.html > > <https://www.mcs.anl.gov/petsc/petsc-3.8.4/docs/manualpages/DMDA/DMDACreate3d.html> > PETSC_COMM_WORLD, & > DM_BOUNDARY_NONE, DM_BOUNDARY_NONE, DM_BOUNDARY_NONE, & > DMDA_STENCIL_BOX, & ! <= stencil type > sum(lx), sum(ly), sum(lz), & ! <= global dimension (of data) in each > direction > px, py, pz, & ! <= number of processors in each > dimension > 1, 0, & ! <= dof per node, stencil width (no > ghost cell) > lx, ly, lz, & ! <= numbers of nodes held by processors > dmobject, ierr ) > > where lx, ly and lz could look like > > lx = (/ 0, 1, 0 /) > ly = (/ 16, 16 /) > lz = (/ 16, 16 /) > > Unfortunately, this does not work: > > >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Argument out of range >> [0]PETSC ERROR: Partition in x direction is too fine! 1 2 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 >> [0]PETSC ERROR: ./prime on a foss_debug named laptwo by bastian Tue Jan 25 >> 23:32:30 2022 >> [0]PETSC ERROR: Configure options PETSC_ARCH=foss_debug --with-cc=gcc >> --with-cxx=g++ --with-fc=gfortran --download-mpich --with-large-file-io=1 >> --with-shared-libraries=1 COPTFLAGS=" " CXXOPTFLAGS=" " FOPTFLAGS=" " >> --march=native --mtune=native >> --download-hypre=/soft/petsc-3.8.4/foss_debug/hypre-v2.12.0.tar.gz >> --with-debugging=yes >> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 318 in >> /soft/petsc-3.8.4/src/dm/impls/da/da3.c >> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in >> /soft/petsc-3.8.4/src/dm/impls/da/dareg.c >> [0]PETSC ERROR: #3 DMSetUp() line 720 in >> /soft/petsc-3.8.4/src/dm/interface/dm.c >> [0]PETSC ERROR: #4 User provided function() line 0 in User file >> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0 > Apparently, lx, ly and lz cannot contain zeros. > (Which would be a useful information in the documentation, too.) > > Is there any workaround? > > My current understanding is that I need to go the extra mile of creating an > additional Communicator involving only those ranks that will share at least > one cell in the subregion DMDA. > > If this is the way to go? If so, how can I control which rank receives which > subdomain in the subregion? DMDACreate3d does not enable me to do so, but I > need to make sure that a rank holds only those cells of the subregion which > are also present in its share of the global domain. > > Many thanks in advance, > Bastian Löhrer > >
