Mark, I configured with Fortran binding enabled with main + MR !4175 <https://gitlab.com/petsc/petsc/-/merge_requests/4175>
'--with-mpiexec=srun', '--with-shared-libraries=1', '--with-cc=cc', '--with-cxx=CC', '--with-fc=ftn', '--with-fortran-bindings', '--with-hip', '--with-hipc=hipcc', '--with-debugging', '--CPPFLAGS=-I${ROCM_PATH}/include', '--CXXPPFLAGS=-I${ROCM_PATH}/include', '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', '--COPTFLAGS=-g -O0', '--CXXOPTFLAGS=-g -O0', '--FOPTFLAGS=-g -O0', '--download-kokkos', '--download-kokkos-kernels', '--download-kokkos-commit=3.4.01', '--download-kokkos-kernels-commit=3.4.01', '--with-kokkos-hip-arch=VEGA908', and 'make check' ran smoothly on a compute node $ make check Running check examples to verify correct installation Using PETSC_DIR=/ccs/home/jczhang/petsc and PETSC_ARCH=arch-spock-cray-kokkos-dbg C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes C/C++ example src/snes/tutorials/ex3k run successfully with kokkos-kernels Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI process Completed test examples --Junchao Zhang On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfad...@lbl.gov> wrote: > And I find that this error, on non-Kokkos C tests, is fixed by turning the > fortran bindings off: > > ld.lld: error: > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > ld.lld: error: > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > ld.lld: error: > /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: > undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > [--no-allow-shlib-undefined] > > On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfad...@lbl.gov> wrote: > >> Not complex. THis has some overlap with my problem w/o Kokkos. >> >> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang <junchao.zh...@gmail.com> >> wrote: >> >>> Do you use complex? post your configure.log. >>> >>> --Junchao Zhang >>> >>> >>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfad...@lbl.gov> wrote: >>> >>>> The simple Kokkos example is failing for me on Spock. >>>> Any ideas? >>>> Thanks, >>>> >>>> 10:44 main *= >>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall -Wwrite-strings >>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>> >>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>> -lquadmath -lstdc++ -ldl -o ex3k >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>> error: no member named 'lock_address_host_space' in namespace >>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply >>>> 'lock_address_host_space'? >>>> while (!Impl::lock_address_host_space((void*)dest)) >>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> lock_address_host_space >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>> note: 'lock_address_host_space' declared here >>>> bool lock_address_host_space(void* ptr); >>>> ^ >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>> did you mean simply 'memory_fence'? >>>> Kokkos::memory_fence(); >>>> ^~~~~~~~~~~~~~~~~~~~ >>>> memory_fence >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>> note: 'memory_fence' declared here >>>> void memory_fence() { >>>> ^ >>>> In file included from ex3k.kokkos.cxx:3: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>> In file included from >>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>> In file included from >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>> error: no member named 'memory_fence' in namespace 'Kokkos::Impl::Kokkos'; >>>> did you mean simply 'memory_fence'? >>>> Kokkos::memory_fence(); >>>> ^~~~~~~~~~~~~~~~~~~~ >>>> memory_fence >>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>> note: 'memory_fence' declared here >>>> void memory_fence() { >>>> ^ >>>> >>>