[petsc-dev] MatPinToCPU
I've tried to add pining the matrix and prolongator to the CPU on coarse grids in GAMG with this: /* pin reduced coase grid - could do something smarter */ ierr = MatPinToCPU(*a_Amat_crs,PETSC_TRUE);CHKERRQ(ierr); ierr = MatPinToCPU(*a_P_inout,PETSC_TRUE);CHKERRQ(ierr); It does not seem to work. It does not look like CUDA has an MatCreateVecs. Should I add one and copy this flag over? Mark
Re: [petsc-dev] valid_GPU_matrix in interface code
Hi Stefano, I have just noticed we have different occurrences of the valid_GPU_matrix flag in src/mat/interface and src/mat/utils I think that how they are used now is wrong, as they assume that all those operations can only be executed on the CPU, irrespective of the specific type. Is there any plan to fix this? If not, I can take care of it, I am currently writing a dense matrix for CUDA here https://bitbucket.org/petsc/petsc/branch/stefano_zampini/cuda there is going to be a major reorganization of the GPU code over the next few weeks, cf. https://bitbucket.org/petsc/petsc/issues/322/working-group-gpus This, however, is pretty much orthogonal to your development of a dense matrix type. Once your branch is ready, it can be rather easily adopted. As for the valid_GPU_matrix flag: This will be fixed alongside the the GPU reorganization. However, it might take 2-3 weeks until I finally get to that part. I guess you will need it sooner? Best regards, Karli
Re: [petsc-dev] Fwd: [mumps-users] MUMPS MPI (5.2.1) does not compile
Ok - I think the current issue is about MUMPS building shared libraries. Current mumps release lacks this feature. And these patches enable building mumps as shared library - but the patch is linux specific - and not portable. It would be great if this can be made portable. Currently when mumps is built via petsc using --download-mumps - its build as static - but with -fPIC [or equivalent] - so it gets sucked into libpetsc.so. [the same with scalapack]. Satish On Mon, 22 Jul 2019, Antonio Trande via petsc-dev wrote: > On 22/07/19 20:28, Pierre Jolivet via petsc-dev wrote: > > We are having trouble compiling git.mumps with similar errors (using > > ifort 2017, yikes). > > Could the forwarded patches be applied to petsc/pkg-mumps, please? > > > > Thanks in advance (and sorry if this should be handled @petsc-maint), > > Pierre > > I have just used 1 Make job (make -j1) to complete the compilation. > Anyway, MUMPS-5.2.1 has been just now pushed on Fedora repositories and > compiled fine on Fedora 31 (and CentOS7): > https://src.fedoraproject.org/rpms/MUMPS/tree/master > > >
Re: [petsc-dev] MatPinToCPU
> On Jul 23, 2019, at 9:12 AM, Mark Adams via petsc-dev
> wrote:
>
> I've tried to add pining the matrix and prolongator to the CPU on coarse
> grids in GAMG with this:
>
> /* pin reduced coase grid - could do something smarter */
> ierr = MatPinToCPU(*a_Amat_crs,PETSC_TRUE);CHKERRQ(ierr);
> ierr = MatPinToCPU(*a_P_inout,PETSC_TRUE);CHKERRQ(ierr);
What are the symptoms of it not working? Does it appear to be still copying
the matrices to the GPU? then running the functions on the GPU?
I suspect the pinning is incompletely done for CUDA (and MPIOpenCL) matrices.
We need the equivalent of
static PetscErrorCode MatPinToCPU_SeqAIJViennaCL(Mat A,PetscBool flg)
{
PetscFunctionBegin;
A->pinnedtocpu = flg;
if (flg) {
A->ops->mult = MatMult_SeqAIJ;
A->ops->multadd= MatMultAdd_SeqAIJ;
A->ops->assemblyend= MatAssemblyEnd_SeqAIJ;
A->ops->duplicate = MatDuplicate_SeqAIJ;
} else {
A->ops->mult = MatMult_SeqAIJViennaCL;
A->ops->multadd= MatMultAdd_SeqAIJViennaCL;
A->ops->assemblyend= MatAssemblyEnd_SeqAIJViennaCL;
A->ops->destroy= MatDestroy_SeqAIJViennaCL;
A->ops->duplicate = MatDuplicate_SeqAIJViennaCL;
}
PetscFunctionReturn(0);
}
for MPIViennaCL and MPISeqAIJ Cusparse but it doesn't look like it has been
written yet.
>
> It does not seem to work. It does not look like CUDA has an MatCreateVecs.
> Should I add one and copy this flag over?
We do need this function. But I don't see how it relates to pinning. When
the matrix is pinned to the CPU we want it to create CPU vectors which I assume
it does.
>
> Mark
Re: [petsc-dev] MatPinToCPU
>
>
> What are the symptoms of it not working? Does it appear to be still
> copying the matrices to the GPU? then running the functions on the GPU?
>
>
The object is dispatching the CUDA mat-vec etc.
I suspect the pinning is incompletely done for CUDA (and MPIOpenCL)
> matrices.
>
>
Yes, git grep MatPinToCPU shows stuff for ViennaCL but not CUDA.
I guess I can add something like this below. Do we need to set the device
methods? They are already set when this method is set, right?
> We need the equivalent of
>
> static PetscErrorCode MatPinToCPU_SeqAIJViennaCL(Mat A,PetscBool flg)
> {
> PetscFunctionBegin;
> A->pinnedtocpu = flg;
> if (flg) {
> A->ops->mult = MatMult_SeqAIJ;
> A->ops->multadd= MatMultAdd_SeqAIJ;
> A->ops->assemblyend= MatAssemblyEnd_SeqAIJ;
> A->ops->duplicate = MatDuplicate_SeqAIJ;
> } else {
> A->ops->mult = MatMult_SeqAIJViennaCL;
> A->ops->multadd= MatMultAdd_SeqAIJViennaCL;
> A->ops->assemblyend= MatAssemblyEnd_SeqAIJViennaCL;
> A->ops->destroy= MatDestroy_SeqAIJViennaCL;
> A->ops->duplicate = MatDuplicate_SeqAIJViennaCL;
> }
> PetscFunctionReturn(0);
> }
>
> for MPIViennaCL and MPISeqAIJ Cusparse but it doesn't look like it has
> been written yet.
>
>
> >
> > It does not seem to work. It does not look like CUDA has an
> MatCreateVecs. Should I add one and copy this flag over?
>
>We do need this function. But I don't see how it relates to pinning.
> When the matrix is pinned to the CPU we want it to create CPU vectors which
> I assume it does.
>
>
> >
> > Mark
>
>
Re: [petsc-dev] MatPinToCPU
Yes, it needs to be able to switch back and forth between the CPU and GPU
methods so you need to move into it the setting of the methods that is
currently directly in the create method. See how
MatConvert_SeqAIJ_SeqAIJViennaCL() calls ierr =
MatPinToCPU_SeqAIJViennaCL(A,PETSC_FALSE);CHKERRQ(ierr); to set the methods for
the GPU initially.
Barry
> On Jul 23, 2019, at 7:32 PM, Mark Adams wrote:
>
>
> What are the symptoms of it not working? Does it appear to be still copying
> the matrices to the GPU? then running the functions on the GPU?
>
>
> The object is dispatching the CUDA mat-vec etc.
>
> I suspect the pinning is incompletely done for CUDA (and MPIOpenCL)
> matrices.
>
>
> Yes, git grep MatPinToCPU shows stuff for ViennaCL but not CUDA.
>
> I guess I can add something like this below. Do we need to set the device
> methods? They are already set when this method is set, right?
>
> We need the equivalent of
>
> static PetscErrorCode MatPinToCPU_SeqAIJViennaCL(Mat A,PetscBool flg)
> {
> PetscFunctionBegin;
> A->pinnedtocpu = flg;
> if (flg) {
> A->ops->mult = MatMult_SeqAIJ;
> A->ops->multadd= MatMultAdd_SeqAIJ;
> A->ops->assemblyend= MatAssemblyEnd_SeqAIJ;
> A->ops->duplicate = MatDuplicate_SeqAIJ;
> } else {
> A->ops->mult = MatMult_SeqAIJViennaCL;
> A->ops->multadd= MatMultAdd_SeqAIJViennaCL;
> A->ops->assemblyend= MatAssemblyEnd_SeqAIJViennaCL;
> A->ops->destroy= MatDestroy_SeqAIJViennaCL;
> A->ops->duplicate = MatDuplicate_SeqAIJViennaCL;
> }
> PetscFunctionReturn(0);
> }
>
> for MPIViennaCL and MPISeqAIJ Cusparse but it doesn't look like it has been
> written yet.
>
>
> >
> > It does not seem to work. It does not look like CUDA has an MatCreateVecs.
> > Should I add one and copy this flag over?
>
>We do need this function. But I don't see how it relates to pinning. When
> the matrix is pinned to the CPU we want it to create CPU vectors which I
> assume it does.
>
>
> >
> > Mark
>
