In theory PETSc Mat have two "state" values,
1) nonzerostate - this is increased anytime the nonzero structure changes
2) state - this is increased anytime any numerical values are changed
These are used by PCSetUp() to determine if the preconditioner needs to be
updated and if that involves a new nonzero structure.
The reason I say "in theory" is that it looks like certain changes to matrices
do not properly update the state.
Hi Barry: Sounds good. In the meantime, I'm simply making a corresponding
_SeqAIJMKL version of all of the functions in question that will just call the
_SeqAIJ version and then call a function that creates an updated MKL sparse
matrix handle. It appears that several of these do not have function prototypes
in any of the .h files (they haven't been needed outside of aij.c). I assume I
can just add PETSC_INTERN prototypes in src/mat/impls/aij/seq/aij.h (where we
have things like prototypes for MatAssemblyEnd_SeqAIJ)?
I think this is the correct place. If they have static in front of them you
will need to remove that also.
I'm just wondering: Isn't fixing the state variables the faster, easier and
more maintainable approach?
So you fix the state variable by reseting with each each function that changes the
matrix entries, how does that magically cause the MKL "convert to MKL format"
in these cases? It is a slightly orthogonal issue I think.
Well, I thought of AIJMKL keeping track of the state for which
mkl_sparse_optimize() was called. If the matrix state changes, the next
call to MatMult() will detect that the current state does not match the
state it was optimized for and hence trigger another optimization. This
isn't too different from what we do for GPU stuff.
Or for every PETSc object do we optionally provide a callback function that is called
with each change to state? Currently this would not be efficient because there may be
several (many) changes to state before we want the subtype data structure to be updated.
At the moment we have a "hard" MatAssemblyEnd() that handles changes in nonzero
structure, may be we need something similar for changes in numerical value? BTW this is
also true for CUDA/CL representations?
A callback for every state change is too greedy, yes. As long as state
and nonzerostate are reliable, we should be good in terms of outside
dependencies on the matrix values and structure, respectively.
Best regards,
Karli
