Hi!
I'm working on a convection-diffusion heat transfer problem. The temperature
is discretized using standard Q1 elements and a DMDA. The flow is modelled
using a stabilized Q1-Q0 method for which DMStag seemed like a good choice. The
codes for the temperature
and flow work fine separately (both in serial and parallel), but when combined
and running
in parallel, a problem sometimes arises in the assembly of the thermal system
matrix.
Here's a rough sketch of the combined code:
// Create dmda for thermal problem and dmstag for flow problem
DMDACreate3d(PETSC_COMM_WORLD, bx, by, bz, DMDA_STENCIL_BOX, nelx+1, nely+1,
nelz+1, PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE,
1, stencilWidth, 0, 0, 0, &dmthermal);
...
// A bit of code to adjust Lx,Ly,Lz so that dmthermal and dmflow are compatible
in the sense of having the same
// local elements
...
DMStagCreate3d(PETSC_COMM_WORLD, bx, by, bz, nelx, nely, nelz, md,nd,pd,
3,0,0,0,DMSTAG_STENCIL_BOX,stencilWidth,Lx,Ly,Lz,&dmflow);
PetscInt edofT[8]; // 8-noded element with 1 temp DOF per node
DMStagStencil edofF[24]; // 8 nodes with 3 velocity DOFs each
// Assemble thermal system matrix K
for (PetscInt e=0 ...) // Loop over local elements
{
// Populate edofF, edofT
// Get element velocities in ue from local
velocity vector uloc
DMStagVecGetValuesStencil(dmflow,uloc,24,edof,ue);
...
Ke = Ke_diffusion + Ke_convection(ue)
...
MatSetValuesLocal(K, 8, edofT, 8, edofT, Ke,
ADD_VALUES);
}
This always works fine in serial, but depending on the mesh and the number of
ranks,
we don't always get the correct values in the element velocity vector ue. I
suspect
this has something to do with the ordering of the elements and/or the DOFs,
because
the elements in the global velocity vector are always the same but their order
may change
(judging from the output of VecView at least).
Is it possible to ensure compatibility between the dm:s, or find some kind of
mapping
between them, so that something along the lines of the code above always works?
Kind regards,
Carl-Johan
