[petsc-dev] DMKSP and pcfieldsplit

[Lawrence Mitchell via petsc-dev]

Dear petsc-dev,

95dbaa6faf01fdfd99114b7c9e5668e4b2aa754d

which does this:

commit 95dbaa6faf01fdfd99114b7c9e5668e4b2aa754d
Author: Matthew G. Knepley <knep...@gmail.com>
Date:   Fri Oct 26 14:12:14 2018 -0400

    PC: FieldSplit must copy DMKSP routines to subDM

diff --git a/src/ksp/pc/impls/fieldsplit/fieldsplit.c 
b/src/ksp/pc/impls/fieldsplit/fieldsplit.c
index b6a84ea342..f43f831204 100644
--- a/src/ksp/pc/impls/fieldsplit/fieldsplit.c
+++ b/src/ksp/pc/impls/fieldsplit/fieldsplit.c
@@ -386,6 +386,13 @@ static PetscErrorCode PCFieldSplitSetDefaults(PC pc)
           ierr = PetscObjectSetOptionsPrefix((PetscObject)(dms[i]), 
prefix);CHKERRQ(ierr);
           ierr = KSPSetDM(ilink->ksp, dms[i]);CHKERRQ(ierr);
           ierr = KSPSetDMActive(ilink->ksp, PETSC_FALSE);CHKERRQ(ierr);
+          {
+            PetscErrorCode (*func)(KSP,Mat,Mat,void*);
+            void            *ctx;
+
+            ierr = DMKSPGetComputeOperators(pc->dm, &func, &ctx);CHKERRQ(ierr);
+            ierr = DMKSPSetComputeOperators(dms[i],  func,  ctx);CHKERRQ(ierr);
+          }
           ierr = 
PetscObjectIncrementTabLevel((PetscObject)dms[i],(PetscObject)ilink->ksp,0);CHKERRQ
           ierr = DMDestroy(&dms[i]);CHKERRQ(ierr);
         }

Breaks many uses of PCMG and fieldsplit for us.

I would like to try and understand how best to fix it. My understanding of the 
way this stuff works is that when I say:

KSPSetComputeOperators

I provide a callback function (which I control), and some user-defined context 
(which I also control).

What this code is doing (which previously it did not), is to transfer this 
callback and context inside PCFieldsplit from a DM defining the full problem 
onto a DM from DMCreateFieldDecomposition (i.e. only part of the problem).

This seems like it is tremendously fragile, because PETSc has no way of knowing 
that the context for the full DM (which is just a void *) is valid for the 
subdm. In particular, in my case, I think it turns out not to be.

What is the right way to think about fixing this? I think that if the DM can 
create a field decomposition, it should also be able to transfer user callbacks 
from the parent to the subdms.

That way, the DM (IOW me) can control how that happens.

Or do I have the way this is supposed to work all wrong. Is the 
ComputeOperators callback I set meant to be completely agnostic, and it should 
get all the problem-specific information from the DM?

Cheers,

Lawrence