Re: [petsc-dev] CUDA GAMG coarse grid solver

[Smith, Barry F. via petsc-dev]

> On Jul 21, 2019, at 8:55 AM, Mark Adams via petsc-dev <petsc-dev@mcs.anl.gov> 
> wrote:
> 
> I am running ex56 with -ex56_dm_vec_type cuda -ex56_dm_mat_type aijcusparse 
> and I see no GPU communication in MatSolve (the serial LU coarse grid solver).

   Do you mean to say, you DO see communication?

   What does -ksp_view should you? It should show the factor type in the 
information about the coarse grid solve?

   You might need something like -mg_coarse_mat_solver_type cusparse  (because 
it may default to the PETSc one, it may be possible to have it default to the 
cusparse if it exists and the matrix is of type MATSEQAIJCUSPARSE).

   The determination of the MatGetFactor() is a bit involved including pasting 
together strings and string compares and could be finding a CPU factorization.

   I could run on one MPI_Rank() in the debugger and put a break point in 
MatGetFactor() and track along to see what it picks and why. You could do this 
debugging without GAMG first, just -pc_type lu  

> GAMG does set the coarse grid solver to LU manually like this: ierr = 
> PCSetType(pc2, PCLU);CHKERRQ(ierr);

  For parallel runs this won't work using the GPU code and only sequential 
direct solvers, so it must using BJACOBI in that case?

   Barry



   

> I am thinking the dispatch of the CUDA version of this got dropped somehow. 
> 
> I see that this is getting called:
> 
> PETSC_EXTERN PetscErrorCode MatSolverTypeRegister_CUSPARSE(void)
> {
>   PetscErrorCode ierr;
> 
>   PetscFunctionBegin;
>   ierr = 
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_LU,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   ierr = 
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_CHOLESKY,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   ierr = 
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_ILU,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   ierr = 
> MatSolverTypeRegister(MATSOLVERCUSPARSE,MATSEQAIJCUSPARSE,MAT_FACTOR_ICC,MatGetFactor_seqaijcusparse_cusparse);CHKERRQ(ierr);
>   PetscFunctionReturn(0);
> }
> 
> but MatGetFactor_seqaijcusparse_cusparse is not getting  called.
> 
> GAMG does set the coarse grid solver to LU manually like this: ierr = 
> PCSetType(pc2, PCLU);CHKERRQ(ierr);
> 
> Any ideas?
> 
> Thanks,
> Mark
> 
>