I'm not sure what to do here. The problem is that pinned-to-cpu vectors are calling *VecCUDACopyFromGPU* here.
Should I set *x->valid_GPU_array *to something else, like PETSC_OFFLOAD_CPU, in PinToCPU so this block of code i s not executed? PetscErrorCode VecGetArray(Vec x,PetscScalar **a) { PetscErrorCode ierr; #if defined(PETSC_HAVE_VIENNACL) PetscBool is_viennacltype = PETSC_FALSE; #endif PetscFunctionBegin; PetscValidHeaderSpecific(x,VEC_CLASSID,1); ierr = VecSetErrorIfLocked(x,1);CHKERRQ(ierr); if (x->petscnative) { #if defined(PETSC_HAVE_VIENNACL) || defined(PETSC_HAVE_CUDA) if (*x->valid_GPU_array* == PETSC_OFFLOAD_GPU) { #if defined(PETSC_HAVE_VIENNACL) ierr = PetscObjectTypeCompareAny((PetscObject)x,&is_viennacltype,VECSEQVIENNACL,VECMPIVIENNACL,VECVIENNACL,"");CHKERRQ(ierr); if (is_viennacltype) { ierr = VecViennaCLCopyFromGPU(x);CHKERRQ(ierr); } else #endif { #if defined(PETSC_HAVE_CUDA) * ierr = VecCUDACopyFromGPU(x);CHKERRQ(ierr);*#endif } } else if (x->valid_GPU_array == PETSC_OFFLOAD_UNALLOCATED) { #if defined(PETSC_HAVE_VIENNACL) ierr = PetscObjectTypeCompareAny((PetscObject)x,&is_viennacltype,VECSEQVIENNACL,VECMPIVIENNACL,VECVIENNACL,"");CHKERRQ(ierr); if (is_viennacltype) { ierr = VecViennaCLAllocateCheckHost(x);CHKERRQ(ierr); } else #endif { #if defined(PETSC_HAVE_CUDA) ierr = VecCUDAAllocateCheckHost(x);CHKERRQ(ierr); #endif } } #endif *a = *((PetscScalar**)x->data); } else { On Tue, Jul 23, 2019 at 9:18 PM Smith, Barry F. <bsm...@mcs.anl.gov> wrote: > > Yes, it needs to be able to switch back and forth between the CPU and GPU > methods so you need to move into it the setting of the methods that is > currently directly in the create method. See how > MatConvert_SeqAIJ_SeqAIJViennaCL() calls ierr = > MatPinToCPU_SeqAIJViennaCL(A,PETSC_FALSE);CHKERRQ(ierr); to set the methods > for the GPU initially. > > Barry > > > > On Jul 23, 2019, at 7:32 PM, Mark Adams <mfad...@lbl.gov> wrote: > > > > > > What are the symptoms of it not working? Does it appear to be still > copying the matrices to the GPU? then running the functions on the GPU? > > > > > > The object is dispatching the CUDA mat-vec etc. > > > > I suspect the pinning is incompletely done for CUDA (and MPIOpenCL) > matrices. > > > > > > Yes, git grep MatPinToCPU shows stuff for ViennaCL but not CUDA. > > > > I guess I can add something like this below. Do we need to set the > device methods? They are already set when this method is set, right? > > > > We need the equivalent of > > > > static PetscErrorCode MatPinToCPU_SeqAIJViennaCL(Mat A,PetscBool flg) > > { > > PetscFunctionBegin; > > A->pinnedtocpu = flg; > > if (flg) { > > A->ops->mult = MatMult_SeqAIJ; > > A->ops->multadd = MatMultAdd_SeqAIJ; > > A->ops->assemblyend = MatAssemblyEnd_SeqAIJ; > > A->ops->duplicate = MatDuplicate_SeqAIJ; > > } else { > > A->ops->mult = MatMult_SeqAIJViennaCL; > > A->ops->multadd = MatMultAdd_SeqAIJViennaCL; > > A->ops->assemblyend = MatAssemblyEnd_SeqAIJViennaCL; > > A->ops->destroy = MatDestroy_SeqAIJViennaCL; > > A->ops->duplicate = MatDuplicate_SeqAIJViennaCL; > > } > > PetscFunctionReturn(0); > > } > > > > for MPIViennaCL and MPISeqAIJ Cusparse but it doesn't look like it has > been written yet. > > > > > > > > > > It does not seem to work. It does not look like CUDA has an > MatCreateVecs. Should I add one and copy this flag over? > > > > We do need this function. But I don't see how it relates to pinning. > When the matrix is pinned to the CPU we want it to create CPU vectors which > I assume it does. > > > > > > > > > > Mark > > > >