The issues related to this were discussed at: https://gitlab.com/petsc/petsc/-/merge_requests/2506#note_283175046
Satish On Mon, 23 Mar 2020, Richard Tran Mills via petsc-dev wrote: > Colleagues, > > I did not notice this, but Junchao's MR, "Directly pass root/leafdata to MPI > in SF when possible" > > https://gitlab.com/petsc/petsc/-/merge_requests/2506 > > that was merged into master over the weekend causes PETSc to error out if > PETSc has been configured with GPU support but the MPI implementation is > "GPU-aware", unless the user has specified "-use_gpu_aware_mpi 0": > > > [0]PETSC ERROR: PETSc is configured with GPU support, but your MPI is > not GPU-aware. For better performance, please use a GPU-aware MPI. > > [0]PETSC ERROR: For IBM Spectrum MPI on OLCF Summit, you may need > jsrun --smpiargs=-gpu. > > [0]PETSC ERROR: For OpenMPI, you need to configure it --with-cuda > (https://www.open-mpi.org/faq/?category=buildcuda) > > [0]PETSC ERROR: For MVAPICH2-GDR, you need to set MV2_USE_CUDA=1 > (http://mvapich.cse.ohio-state.edu/userguide/gdr/) > > [0]PETSC ERROR: For Cray-MPICH, you need to set > MPICH_RDMA_ENABLED_CUDA=1 > (https://www.olcf.ornl.gov/tutorials/gpudirect-mpich-enabled-cuda/) > > [0]PETSC ERROR: If you do not care, use option -use_gpu_aware_mpi 0, > then PETSc will copy data from GPU to CPU for communication. > > application called MPI_Abort(MPI_COMM_WORLD, 90693076) - process 0 > > I like that we are warning users about a potential performance problem, but > this seems like something that should print a warning, rather than exiting > with an error. So I am wondering > > 1) Do people agree that this should be a warning instead of an error? > > and > > 2) Shouldn't we add a standard mechanism for reporting these sorts of warnings > at runtime? > > --Richard > >