This is probably kokkos pulling in the dependencies where compiling kokkos source within PETSc.
Il Dom 18 Lug 2021, 16:29 Mark Adams <mfad...@lbl.gov> ha scritto: > Whoops, this error was just from not telling gfortran to allow long lines. > > Anway, I did find that the when fortran bindings are enabled this OMP > thing gets into the library. No idea how. I'll askm ORNL. > > 10:26 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc2$ nm > /gpfs/alpine/csc314/scratch/adams/petsc2/arch-spock-opt-cray-kokkos/lib/libpetsc.so > |g offload > U .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa > U .omp_offloading.img_size.cray_amdgcn-amd-amdhsa > U .omp_offloading.img_start.cray_amdgcn-amd-amdhsa > 0000000001d457b0 T vecgetoffloadmask_ > > > On Sun, Jul 18, 2021 at 8:43 AM Mark Adams <mfad...@lbl.gov> wrote: > >> Ah, your test was not on Spock ... I have something working but this is >> strange. >> >> I switched to the GNU ProgEnv. and it passed the Fortran test in 'check', >> but this failed: >> >> >> >> >> >> >> >> >> >> *08:26 2 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >> PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu >> PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >> -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 >> >> -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >> -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >> <http://21.04.1.1/GNU/9.1/x86_64/lib> >> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib >> <http://21.04.1.1/GNU/9.1/x86_64/lib> >> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib >> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >> -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 >> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib >> -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc >> -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >> -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm >> -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml >> -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o >> ex40f90ex40f90.F90:35:132: 35 | call >> DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, >> & | >> 1Error: >> Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* >> * ....* >> >> But ex5f (the Fortran check test) does work manually also (sanity check). >> >> Oddly I seem to have the cray compilers: >> >> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn >> /opt/cray/pe/craype/2.7.6/bin/ftn >> >> yet it is GNU: >> >> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version >> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >> Copyright (C) 2020 Free Software Foundation, Inc. >> This is free software; see the source for copying conditions. There is NO >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> PURPOSE. >> >> The documentation says ftn should be here: >> >> 08:34 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ >> $GCC_PATH/bin/gfortran --version >> GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) >> Copyright (C) 2020 Free Software Foundation, Inc. >> This is free software; see the source for copying conditions. There is NO >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> PURPOSE. >> >> Same, but the path is not the same: >> >> 08:39 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH >> /opt/gcc/10.2.0 >> >> Anyway, thanks for your help, >> Mark >> >> On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zh...@gmail.com> >> wrote: >> >>> Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray >>> compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this >>> error. >>> Probably it is a problem of Spock. But I am not sure. >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfad...@lbl.gov> wrote: >>> >>>> And I can run a fortran test, with warnings, but C tests fail: >>>> >>>> 11:15 jczhang/fix-cray-mpicxx-includes/main= >>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test >>>> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' >>>> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>> PETSC_ARCH=arch-spock-opt-cray-kokkos >>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>> FC >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>> FLINKER >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 >>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>> warning: alignment 128 of symbol >>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>>> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >>>> TEST >>>> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts >>>> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 >>>> >>>> >>>> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfad...@lbl.gov> wrote: >>>> >>>>> HUmm, I can not reproduce this. >>>>> >>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>> PETSC_ARCH=arch-spock-dbg-kokkos check >>>>> Running check examples to verify correct installation >>>>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>>>> PETSC_ARCH=arch-spock-dbg-kokkos >>>>> gmake[3]: >>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>>> ex19.PETSc] Error 2 (ignored) >>>>> *******************Error detected during compile or >>>>> link!******************* >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >>>>> >>>>> ********************************************************************************* >>>>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments >>>>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings >>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>> -Qunused-arguments -fvisibility=hidden -g -O0 >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >>>>> >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/pmi/6.0.10/lib >>>>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>> -lquadmath -lstdc++ -ldl -o ex19 >>>>> ld.lld: error: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> clang-11: error: linker command failed with exit code 1 (use -v to see >>>>> invocation) >>>>> gmake[4]: *** [<builtin>: ex19] Error 1 >>>>> gmake[3]: >>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>>>> ex3k.PETSc] Error 2 (ignored) >>>>> *******************Error detected during compile or >>>>> link!******************* >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >>>>> >>>>> ********************************************************************************* >>>>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>>>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >>>>> 21.04.1.1/CRAY/9.0/x86_64/include >>>>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >>>>> -I/opt/cray/pe/pmi/6.0.10/include >>>>> -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >>>>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >>>>> -Wl,--as-needed,-lsci_cray,--no-as-needed >>>>> -Wl,--as-needed,-lmpi_cray,--no-as-needed >>>>> -Wl,--as-needed,-ldsmml,--no-as-needed >>>>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >>>>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >>>>> ex3k.kokkos.cxx >>>>> >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/pmi/6.0.10/lib >>>>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >>>>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>>>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >>>>> clang-12: error: linker command failed with exit code 1 (use -v to see >>>>> invocation) >>>>> gmake[4]: *** >>>>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >>>>> Error 1 >>>>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >>>>> 1,25c1,4 >>>>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>>>> < Vec Object: Exact Solution 2 MPI processes >>>>> < type: mpikokkos >>>>> < Process [0] >>>>> < 0. >>>>> < 0.015625 >>>>> < 0.125 >>>>> < Process [1] >>>>> < 0.421875 >>>>> < 1. >>>>> < Vec Object: Forcing function 2 MPI processes >>>>> < type: mpikokkos >>>>> < Process [0] >>>>> < 1e-72 >>>>> < 1.50024 >>>>> < 3.01563 >>>>> < Process [1] >>>>> < 4.67798 >>>>> < 7. >>>>> < 0 SNES Function norm 5.414682427127e+00 >>>>> < 1 SNES Function norm 2.952582418265e-01 >>>>> < 2 SNES Function norm 4.502293658739e-04 >>>>> < 3 SNES Function norm 1.389665806646e-09 >>>>> < Number of SNES iterations = 3 >>>>> < Norm of error 1.49752e-10 Iterations 3 >>>>> --- >>>>> > slurmstepd: error: execve(): >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>>> file or directory >>>>> > slurmstepd: error: execve(): >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>>>> file or directory >>>>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >>>>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>>>> Possible problem with ex3k running with kokkos-kernels, diffs above >>>>> ========================================= >>>>> *******************Error detected during compile or >>>>> link!******************* >>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >>>>> ********************************************************* >>>>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >>>>> >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>> -L/opt/cray/pe/pmi/6.0.10/lib >>>>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>> -lquadmath -lstdc++ -ldl -o ex5f >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>> warning: alignment 128 of symbol >>>>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 >>>>> in >>>>> /tmp/pe_30429/ex5f_1.o >>>>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>>>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>>>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 >>>>> in >>>>> /tmp/pe_30429/ex5f_1.o >>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>> process >>>>> Completed test examples >>>>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>>>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >>>>> >>>>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang < >>>>> junchao.zh...@gmail.com> wrote: >>>>> >>>>>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>>>>> >>>>>> '--with-mpiexec=srun', >>>>>> '--with-shared-libraries=1', >>>>>> '--with-cc=cc', >>>>>> '--with-cxx=CC', >>>>>> '--with-fc=ftn', >>>>>> '--with-fortran-bindings', >>>>>> '--with-hip', >>>>>> '--with-hipc=hipcc', >>>>>> '--with-debugging', >>>>>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>>>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>>>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>>>> '--COPTFLAGS=-g -O0', >>>>>> '--CXXOPTFLAGS=-g -O0', >>>>>> '--FOPTFLAGS=-g -O0', >>>>>> '--download-kokkos', >>>>>> '--download-kokkos-kernels', >>>>>> '--download-kokkos-commit=3.4.01', >>>>>> '--download-kokkos-kernels-commit=3.4.01', >>>>>> '--with-kokkos-hip-arch=VEGA908', >>>>>> >>>>>> and 'make check' ran smoothly on a compute node >>>>>> >>>>>> $ make check >>>>>> Running check examples to verify correct installation >>>>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>>>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>>>> process >>>>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>>>> processes >>>>>> C/C++ example src/snes/tutorials/ex3k run successfully with >>>>>> kokkos-kernels >>>>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>>>> process >>>>>> >>>>>> Completed test examples >>>>>> >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfad...@lbl.gov> wrote: >>>>>> >>>>>>> And I find that this error, on non-Kokkos C tests, is fixed by >>>>>>> turning the fortran bindings off: >>>>>>> >>>>>>> ld.lld: error: >>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>>>> [--no-allow-shlib-undefined] >>>>>>> ld.lld: error: >>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>>>> [--no-allow-shlib-undefined] >>>>>>> ld.lld: error: >>>>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>>>> [--no-allow-shlib-undefined] >>>>>>> >>>>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfad...@lbl.gov> wrote: >>>>>>> >>>>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>>>>> >>>>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>>>>> junchao.zh...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Do you use complex? post your configure.log. >>>>>>>>> >>>>>>>>> --Junchao Zhang >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfad...@lbl.gov> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>>>>> Any ideas? >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> 10:44 main *= >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>>>>>>>>> -lhsa-runtime64 >>>>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall >>>>>>>>>> -Wwrite-strings >>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>>>>> >>>>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 >>>>>>>>>> -lhsa-runtime64 >>>>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray >>>>>>>>>> -lmpi_cray >>>>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath >>>>>>>>>> -lcrayacc_amdgpu >>>>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran >>>>>>>>>> -lpthread >>>>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply >>>>>>>>>> 'lock_address_host_space'? >>>>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> lock_address_host_space >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>>>>> note: 'lock_address_host_space' declared here >>>>>>>>>> bool lock_address_host_space(void* ptr); >>>>>>>>>> ^ >>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>>>>> error: no member named 'memory_fence' in namespace >>>>>>>>>> 'Kokkos::Impl::Kokkos'; >>>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>>> Kokkos::memory_fence(); >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> memory_fence >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>>> note: 'memory_fence' declared here >>>>>>>>>> void memory_fence() { >>>>>>>>>> ^ >>>>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>>>> In file included from >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>>>>> error: no member named 'memory_fence' in namespace >>>>>>>>>> 'Kokkos::Impl::Kokkos'; >>>>>>>>>> did you mean simply 'memory_fence'? >>>>>>>>>> Kokkos::memory_fence(); >>>>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>>>> memory_fence >>>>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>>>> note: 'memory_fence' declared here >>>>>>>>>> void memory_fence() { >>>>>>>>>> ^ >>>>>>>>>> >>>>>>>>>