Re: [petsc-dev] kokkos error

2021-11-21 Thread Junchao Zhang 
You can add -v to the offending command line to see what happened, i.e.,
how nvcc_wrapper passed options to g++.

--Junchao Zhang


On Sun, Nov 21, 2021 at 12:05 PM Mark Adams  wrote:

> Any idea what is going wrong with this?
>
> Using PETSC_DIR=/global/homes/m/madams/petsc and
> PETSC_ARCH=arch-perlmutter-opt-gcc-kokkos-cuda
> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
> C/C++ example src/snes/tutorials/ex19 run successfully with cuda
> gmake[3]: [/global/homes/m/madams/petsc/lib/petsc/conf/rules:301:
> ex3k.PETSc] Error 2 (ignored)
> ***Error detected during compile or
> link!***
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /global/homes/m/madams/petsc/src/snes/tutorials ex3k
>
> *
>
> PATH=/global/homes/m/madams/petsc/arch-perlmutter-opt-gcc-kokkos-cuda/lib:/global/common/software/nersc/shasta2105/arm-forge/21.0.2-linux-x86_64/bin:/global/common/software/nersc/cos1.3/cuda/11.3.0/bin:/opt/cray/pe/perftools/21.10.0/bin:/opt/cray/pe/papi/
> 6.0.0.10/bin:/opt/cray/pe/gcc/9.3.0/bin:/opt/cray/pe/craype/2.7.11/bin:/global/common/software/nersc/shasta2105/python/3.8-anaconda-2021.05/bin:/global/homes/m/madams/.local/perlmutter/3.8-anaconda-2021.05/bin:/global/common/software/nersc/shasta2105/cmake/git-master/bin:/global/common/software/nersc/bin:/opt/cray/libfabric/1.11.0.4.79/bin:/usr/local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/opt/cray/pe/bin:`dirname
> 
> /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc`
> NVCC_WRAPPER_DEFAULT_COMPILER=CC
> /global/homes/m/madams/petsc/arch-perlmutter-opt-gcc-kokkos-cuda/bin/nvcc_wrapper
> --expt-extended-lambda -g -Xcompiler -rdynamic -DLANDAU_DIM=2
> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -ccbin CC
> -std=c++17 -gencode arch=compute_80,code=sm_80  -Wno-deprecated-gpu-targets
> -I/opt/cray/pe/mpich/8.1.10/ofi/gnu/9.1/include -I/opt/cray/pe/libsci/
> 21.08.1.2/GNU/9.1/x86_64/include -I/opt/cray/pe/pmi/6.0.14/include
> -I/opt/cray/pe/dsmml/0.2.2/dsmml//include
> -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include
>  -I/global/homes/m/madams/petsc/include
> -I/global/homes/m/madams/petsc/arch-perlmutter-opt-gcc-kokkos-cuda/include
> -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include-g
> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -fPIC -O3 -g
> -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 -O3
> -L/opt/cray/pe/mpich/8.1.10/ofi/gnu/9.1/lib
> -L/opt/cray/pe/mpich/8.1.10/gtl/lib -L/opt/cray/pe/libsci/
> 21.08.1.2/GNU/9.1/x86_64/lib -L/opt/cray/pe/pmi/6.0.14/lib
> -L/opt/cray/pe/dsmml/0.2.2/dsmml//lib
> -L/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/lib64
> -Wl,--as-needed,-lmpi_gnu_91,--no-as-needed
> -Wl,--as-needed,-lsci_gnu_82_mpi,--no-as-needed
> -Wl,--as-needed,-lsci_gnu_82,--no-as-needed
> -Wl,--as-needed,-ldsmml,--no-as-needed  -lpmi -lpmi2 -lmpi_gtl_cuda -ldl
> -lxpmem   ex3k.kokkos.cxx
>  
> -Wl,-rpath,/global/homes/m/madams/petsc/arch-perlmutter-opt-gcc-kokkos-cuda/lib
> -L/global/homes/m/madams/petsc/arch-perlmutter-opt-gcc-kokkos-cuda/lib
> -Wl,-rpath,/global/homes/m/madams/petsc/arch-perlmutter-opt-gcc-kokkos-cuda/lib
> -L/global/homes/m/madams/petsc/arch-perlmutter-opt-gcc-kokkos-cuda/lib
> -Wl,-rpath,/global/common/software/nersc/cos1.3/cuda/11.3.0/lib64
> -L/global/common/software/nersc/cos1.3/cuda/11.3.0/lib64
> -L/global/common/software/nersc/cos1.3/cuda/11.3.0/lib64/stubs
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/ofi/gnu/9.1/lib
> -L/opt/cray/pe/mpich/8.1.10/ofi/gnu/9.1/lib
> -Wl,-rpath,/opt/cray/pe/mpich/8.1.10/gtl/lib
> -L/opt/cray/pe/mpich/8.1.10/gtl/lib -Wl,-rpath,/opt/cray/pe/libsci/
> 21.08.1.2/GNU/9.1/x86_64/lib -L/opt/cray/pe/libsci/
> 21.08.1.2/GNU/9.1/x86_64/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.14/lib
> -L/opt/cray/pe/pmi/6.0.14/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.2.2/dsmml/lib
> -L/opt/cray/pe/dsmml/0.2.2/dsmml/lib
> -Wl,-rpath,/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/lib64
> -L/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/lib64
> -Wl,-rpath,/opt/cray/pe/gcc/9.3.0/snos/lib/gcc/x86_64-suse-linux/9.3.0
> -L/opt/cray/pe/gcc/9.3.0/snos/lib/gcc/x86_64-suse-linux/9.3.0
> -Wl,-rpath,/opt/cray/pe/gcc/9.3.0/snos/lib64
> -L/opt/cray/pe/gcc/9.3.0/snos/lib64
> -Wl,-rpath,/opt/cray/pe/gcc/9.3.0/snos/lib
> -L/opt/cray/pe/gcc/9.3.0/snos/lib -lpetsc -lkokkoskernels
> -lkokkoscontainers -lkokkoscore -lp4est -lsc -lparmetis -lmetis

Re: [petsc-dev] Kokkos error on SUMMIT

2020-10-02 Thread Junchao Zhang 
On Fri, Oct 2, 2020 at 3:02 PM Junchao Zhang 
wrote:

>
>
> On Fri, Oct 2, 2020 at 2:59 PM Mark Adams  wrote:
>
>>
>>
>> On Fri, Oct 2, 2020 at 3:15 PM Barry Smith  wrote:
>>
>>>
>>>   Mark,
>>>
>>>   Looks like you are building Kokkos without CUDA.
>>
>>
>> Yes. This is a CPU build of Kokkos.
>>
>>
>>> You don't have --with-cuda on configure line that is used by Kokkos to
>>> determine what version to build.
>>>
>>>   Junchao,
>>>
>>>   I guess you need to test Kokkos Kernels without CUDA and HIP and make
>>> a few changes.
>>>
>>
>> I'm trying with OpenMP right now. If you want CPU runs then asking for
>> OMP is not terrible.
>>
>>
> That is an interesting feature of Kokkos.
>
By the design, even 'device' memory is now the host memory. The computation
is still diverted to Kokkos backend.  In other words, petsc supports
multithreading through Kokkos.


>
>
>>
>>>   Barry
>>>
>>>
>>>
>>>
>>> #if defined(PETSC_HAVE_CUDA)
>>>   #define WaitForKokkos() PetscCUDASynchronize ? (Kokkos::fence(),0) : 0;
>>> #elif defined(PETSC_HAVE_HIP)
>>>   #define WaitForKokkos() PetscHIPSynchronize ? (Kokkos::fence(),0) : 0;
>>> #endif
>>>
>>>
>>>
>>> > On Oct 2, 2020, at 11:47 AM, Mark Adams  wrote:
>>> >
>>> >
>>> > 
>>>
>>>

Re: [petsc-dev] Kokkos error on SUMMIT

2020-10-02 Thread Junchao Zhang 
On Fri, Oct 2, 2020 at 2:59 PM Mark Adams  wrote:

>
>
> On Fri, Oct 2, 2020 at 3:15 PM Barry Smith  wrote:
>
>>
>>   Mark,
>>
>>   Looks like you are building Kokkos without CUDA.
>
>
> Yes. This is a CPU build of Kokkos.
>
>
>> You don't have --with-cuda on configure line that is used by Kokkos to
>> determine what version to build.
>>
>>   Junchao,
>>
>>   I guess you need to test Kokkos Kernels without CUDA and HIP and make a
>> few changes.
>>
>
> I'm trying with OpenMP right now. If you want CPU runs then asking for OMP
> is not terrible.
>
>
That is an interesting feature of Kokkos.


>
>>   Barry
>>
>>
>>
>>
>> #if defined(PETSC_HAVE_CUDA)
>>   #define WaitForKokkos() PetscCUDASynchronize ? (Kokkos::fence(),0) : 0;
>> #elif defined(PETSC_HAVE_HIP)
>>   #define WaitForKokkos() PetscHIPSynchronize ? (Kokkos::fence(),0) : 0;
>> #endif
>>
>>
>>
>> > On Oct 2, 2020, at 11:47 AM, Mark Adams  wrote:
>> >
>> >
>> > 
>>
>>

Re: [petsc-dev] Kokkos error on SUMMIT

2020-10-02 Thread Mark Adams 
On Fri, Oct 2, 2020 at 3:15 PM Barry Smith  wrote:

>
>   Mark,
>
>   Looks like you are building Kokkos without CUDA.


Yes. This is a CPU build of Kokkos.


> You don't have --with-cuda on configure line that is used by Kokkos to
> determine what version to build.
>
>   Junchao,
>
>   I guess you need to test Kokkos Kernels without CUDA and HIP and make a
> few changes.
>

I'm trying with OpenMP right now. If you want CPU runs then asking for OMP
is not terrible.


>
>   Barry
>
>
>
>
> #if defined(PETSC_HAVE_CUDA)
>   #define WaitForKokkos() PetscCUDASynchronize ? (Kokkos::fence(),0) : 0;
> #elif defined(PETSC_HAVE_HIP)
>   #define WaitForKokkos() PetscHIPSynchronize ? (Kokkos::fence(),0) : 0;
> #endif
>
>
>
> > On Oct 2, 2020, at 11:47 AM, Mark Adams  wrote:
> >
> >
> > 
>
>

Re: [petsc-dev] Kokkos error on SUMMIT

2020-10-02 Thread Barry Smith 


  Mark,

  Looks like you are building Kokkos without CUDA.You don't have 
--with-cuda on configure line that is used by Kokkos to determine what version 
to build.

  Junchao,

  I guess you need to test Kokkos Kernels without CUDA and HIP and make a few 
changes.

  Barry




#if defined(PETSC_HAVE_CUDA)
  #define WaitForKokkos() PetscCUDASynchronize ? (Kokkos::fence(),0) : 0;
#elif defined(PETSC_HAVE_HIP)
  #define WaitForKokkos() PetscHIPSynchronize ? (Kokkos::fence(),0) : 0;
#endif



> On Oct 2, 2020, at 11:47 AM, Mark Adams  wrote:
> 
> 
>