[petsc-users] DMDA with 0 in lx, ly, lz ... or how to create a DMDA for a subregion

[Bastian Löhrer]

  
  
    Dear PETSc community,
    in our CFD code we use a 3D DMDA to organize our data.
    Now I need to compute a derived quantity in a subregion of the
      global domain and to write these data to disk for further
      post-processing.
    The subregion is actually a planar slice for now, but it could
      also be a boxed-shaped region in future applications.
    Hence, I figured I would create a new DMDA for this subregion by
      writing something along the lines of 
    
    call DMDACreate3d(                 & !
https://www.mcs.anl.gov/petsc/petsc-3.8.4/docs/manualpages/DMDA/DMDACreate3d.html
           PETSC_COMM_WORLD,               &
           DM_BOUNDARY_NONE,  DM_BOUNDARY_NONE,  DM_BOUNDARY_NONE, &
           DMDA_STENCIL_BOX,               & ! <= stencil type
           sum(lx), sum(ly), sum(lz),      & ! <= global
        dimension (of data) in each direction
           px, py, pz,                     & ! <= number of
        processors in each dimension
           1, 0,                           & ! <= dof per node,
        stencil width (no ghost cell)
           lx, ly, lz,                     & ! <= numbers of
        nodes held by processors
           dmobject, ierr )
      
    where lx, ly and lz could look like
    
    lx = (/  0,  1,  0 /)
        ly = (/ 16, 16 /)
        lz = (/ 16, 16 /)
      
    Unfortunately, this does not work:
    
      [0]PETSC ERROR: ---------------------
          Error Message
          --------------------------------------------------------------
        [0]PETSC ERROR: Argument out of range
        [0]PETSC ERROR: Partition in x direction is too fine! 1
          2
        [0]PETSC ERROR: See
          http://www.mcs.anl.gov/petsc/documentation/faq.html for
          trouble shooting.
        [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24,
          2018 
        [0]PETSC ERROR: ./prime on a foss_debug named laptwo by
          bastian Tue Jan 25 23:32:30 2022
        [0]PETSC ERROR: Configure options PETSC_ARCH=foss_debug
          --with-cc=gcc --with-cxx=g++ --with-fc=gfortran
          --download-mpich --with-large-file-io=1
          --with-shared-libraries=1 COPTFLAGS="   " CXXOPTFLAGS=" "
          FOPTFLAGS="   " --march=native --mtune=native
          --download-hypre=/soft/petsc-3.8.4/foss_debug/hypre-v2.12.0.tar.gz
          --with-debugging=yes
        [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 318 in
          /soft/petsc-3.8.4/src/dm/impls/da/da3.c
        [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in
          /soft/petsc-3.8.4/src/dm/impls/da/dareg.c
        [0]PETSC ERROR: #3 DMSetUp() line 720 in
          /soft/petsc-3.8.4/src/dm/interface/dm.c
        [0]PETSC ERROR: #4 User provided function() line 0 in
          User file
        application called MPI_Abort(MPI_COMM_WORLD, 63) -
          process 0
      
      Apparently, lx, ly and lz cannot
      contain zeros.
      (Which would be a useful information in the documentation, too.)
    Is there any workaround?
    
    My current understanding is that I need to go the extra mile of
      creating an additional Communicator involving only those ranks
      that will share at least one cell in the subregion DMDA.
    If this is the way to go? If so, how can I control which rank
      receives which subdomain in the subregion?  DMDACreate3d does not
      enable me to do so, but I need to make sure that a rank holds only
      those cells of the subregion which are also present in its share
      of the global domain.
    Many thanks in advance,
      Bastian Löhrer