Ah, your test was not on Spock ... I have something working but this is strange.
I switched to the GNU ProgEnv. and it passed the Fortran test in 'check', but this failed: *08:26 2 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make PETSC_DIR=/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu PETSC_ARCH="" ex40f90ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 -I/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/include -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex40f90.F90 -Wl,-rpath,/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib -L/gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-dbg-gnu/lib -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib -L/opt/cray/pe/mpich/8.1.4/ofi/gnu/9.1/lib -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib <http://21.04.1.1/GNU/9.1/x86_64/lib> -L/opt/cray/pe/libsci/21.04.1.1/GNU/9.1/x86_64/lib <http://21.04.1.1/GNU/9.1/x86_64/lib> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 -Wl,-rpath,/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 -L/opt/gcc/10.2.0/snos/lib/gcc/x86_64-suse-linux/10.2.0 -Wl,-rpath,/opt/gcc/10.2.0/snos/lib64 -L/opt/gcc/10.2.0/snos/lib64 -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib -L/sw/spock/spack-envs/views/rocm-4.1.0/llvm/lib -Wl,-rpath,/opt/gcc/10.2.0/snos/lib -L/opt/gcc/10.2.0/snos/lib -lpetsc -lhipsparse -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 -lstdc++ -ldl -lpmi -lmpi_gtl_hsa -lxpmem -lgfortran -lm -lsci_gnu_82_mpi -lsci_gnu_82 -lmpifort_gnu_91 -lmpi_gnu_91 -ldsmml -lgfortran -lquadmath -lpthread -lm -lgcc_s -lquadmath -lstdc++ -ldl -o ex40f90ex40f90.F90:35:132: 35 | call DMDACreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_BOX,ten,ten,PETSC_DECIDE,PETSC_DECIDE,two,one, & | 1Error: Line truncated at (1) [-Werror=line-truncation]ex40f90.F90:35:132:* * ....* But ex5f (the Fortran check test) does work manually also (sanity check). Oddly I seem to have the cray compilers: 08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ which ftn /opt/cray/pe/craype/2.7.6/bin/ftn yet it is GNU: 08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ ftn --version GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. The documentation says ftn should be here: 08:34 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ $GCC_PATH/bin/gfortran --version GNU Fortran (GCC) 10.2.0 20200723 (Cray Inc.) Copyright (C) 2020 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Same, but the path is not the same: 08:39 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ echo $GCC_PATH /opt/gcc/10.2.0 Anyway, thanks for your help, Mark On Sat, Jul 17, 2021 at 11:26 PM Junchao Zhang <junchao.zh...@gmail.com> wrote: > Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray > compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this > error. > Probably it is a problem of Spock. But I am not sure. > > --Junchao Zhang > > > On Sat, Jul 17, 2021 at 10:17 AM Mark Adams <mfad...@lbl.gov> wrote: > >> And I can run a fortran test, with warnings, but C tests fail: >> >> 11:15 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test >> search='ts_utils_dmplexlandau_tutorials-ex1f90_0' >> Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 >> PETSC_ARCH=arch-spock-opt-cray-kokkos >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> FC >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> FLINKER >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 128 of symbol >> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >> arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o >> TEST >> arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts >> ok ts_utils_dmplexlandau_tutorials-ex1f90_0 >> ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 >> >> >> On Sat, Jul 17, 2021 at 10:53 AM Mark Adams <mfad...@lbl.gov> wrote: >> >>> HUmm, I can not reproduce this. >>> >>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc$ make >>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>> PETSC_ARCH=arch-spock-dbg-kokkos check >>> Running check examples to verify correct installation >>> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >>> PETSC_ARCH=arch-spock-dbg-kokkos >>> gmake[3]: >>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>> ex19.PETSc] Error 2 (ignored) >>> *******************Error detected during compile or >>> link!******************* >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >>> >>> ********************************************************************************* >>> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>> -lhsa-runtime64 -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>> -Wno-unknown-pragmas -fstack-protector -Qunused-arguments >>> -fvisibility=hidden -g -O0 -fPIC -Wall -Wwrite-strings >>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>> -Qunused-arguments -fvisibility=hidden -g -O0 >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex19.c >>> >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex19 >>> ld.lld: error: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> ld.lld: error: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> ld.lld: error: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>> [--no-allow-shlib-undefined] >>> clang-11: error: linker command failed with exit code 1 (use -v to see >>> invocation) >>> gmake[4]: *** [<builtin>: ex19] Error 1 >>> gmake[3]: >>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >>> ex3k.PETSc] Error 2 (ignored) >>> *******************Error detected during compile or >>> link!******************* >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex3k >>> >>> ********************************************************************************* >>> hipcc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing >>> -Wno-unknown-pragmas -fvisibility=hidden -g -O0 -std=c++14 >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include -I/opt/cray/pe/libsci/ >>> 21.04.1.1/CRAY/9.0/x86_64/include >>> -I/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/include >>> -I/opt/cray/pe/pmi/6.0.10/include -I/opt/cray/pe/dsmml/0.1.4/dsmml//include >>> -I/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/include >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -L/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml//lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib/pkgconfig/../ >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,--as-needed,-lsci_cray_mpi,--no-as-needed >>> -Wl,--as-needed,-lsci_cray,--no-as-needed >>> -Wl,--as-needed,-lmpi_cray,--no-as-needed >>> -Wl,--as-needed,-ldsmml,--no-as-needed >>> -Wl,--as-needed,-lpgas-shmem,--no-as-needed -lpmi -ldl -lmpi_gtl_hsa >>> -lxpmem -lquadmath -lmodules -lfi -lcraymath -lf -lu -lcsup >>> ex3k.kokkos.cxx >>> >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex3k >>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to `.omp_offloading.img_size.cray_amdgcn-amd-amdhsa' >>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to `.omp_offloading.img_start.cray_amdgcn-amd-amdhsa' >>> /usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: >>> /gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib/libpetsc.so: >>> undefined reference to `.omp_offloading.img_cache.cray_amdgcn-amd-amdhsa' >>> clang-12: error: linker command failed with exit code 1 (use -v to see >>> invocation) >>> gmake[4]: *** >>> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/test:26: ex3k] >>> Error 1 >>> gmake[3]: [makefile:102: runex3k_kokkos] Error 2 (ignored) >>> 1,25c1,4 >>> < atol=1e-50, rtol=1e-08, stol=1e-08, maxit=50, maxf=10000 >>> < Vec Object: Exact Solution 2 MPI processes >>> < type: mpikokkos >>> < Process [0] >>> < 0. >>> < 0.015625 >>> < 0.125 >>> < Process [1] >>> < 0.421875 >>> < 1. >>> < Vec Object: Forcing function 2 MPI processes >>> < type: mpikokkos >>> < Process [0] >>> < 1e-72 >>> < 1.50024 >>> < 3.01563 >>> < Process [1] >>> < 4.67798 >>> < 7. >>> < 0 SNES Function norm 5.414682427127e+00 >>> < 1 SNES Function norm 2.952582418265e-01 >>> < 2 SNES Function norm 4.502293658739e-04 >>> < 3 SNES Function norm 1.389665806646e-09 >>> < Number of SNES iterations = 3 >>> < Norm of error 1.49752e-10 Iterations 3 >>> --- >>> > slurmstepd: error: execve(): >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>> file or directory >>> > slurmstepd: error: execve(): >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials/./ex3k: No such >>> file or directory >>> > srun: error: spock01: tasks 0-1: Exited with exit code 2 >>> > srun: launch/slurm: _step_signal: Terminating StepId=272979.0 >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials >>> Possible problem with ex3k running with kokkos-kernels, diffs above >>> ========================================= >>> *******************Error detected during compile or >>> link!******************* >>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex5f >>> ********************************************************* >>> ftn -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 >>> -lhsa-runtime64 -fPIC -g -O0 -fPIC -g -O0 >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex5f.F90 >>> >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>> -lquadmath -lstdc++ -ldl -o ex5f >>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>> warning: alignment 128 of symbol >>> `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in >>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>> /tmp/pe_30429/ex5f_1.o >>> /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: >>> warning: alignment 64 of symbol `$data_init$iso_c_binding_' in >>> /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in >>> /tmp/pe_30429/ex5f_1.o >>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>> process >>> Completed test examples >>> 10:49 jczhang/fix-cray-mpicxx-includes/main= >>> /gpfs/alpine/csc314/scratch/adams/petsc$ ll >>> >>> On Fri, Jul 16, 2021 at 11:28 PM Junchao Zhang <junchao.zh...@gmail.com> >>> wrote: >>> >>>> Mark, I configured with Fortran binding enabled with main + MR !4175 >>>> <https://gitlab.com/petsc/petsc/-/merge_requests/4175> >>>> >>>> '--with-mpiexec=srun', >>>> '--with-shared-libraries=1', >>>> '--with-cc=cc', >>>> '--with-cxx=CC', >>>> '--with-fc=ftn', >>>> '--with-fortran-bindings', >>>> '--with-hip', >>>> '--with-hipc=hipcc', >>>> '--with-debugging', >>>> '--CPPFLAGS=-I${ROCM_PATH}/include', >>>> '--CXXPPFLAGS=-I${ROCM_PATH}/include', >>>> '--CC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>> '--CXX_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>> '--FC_LINKER_FLAGS=-L${ROCM_PATH}/lib -lamdhip64 -lhsa-runtime64', >>>> '--COPTFLAGS=-g -O0', >>>> '--CXXOPTFLAGS=-g -O0', >>>> '--FOPTFLAGS=-g -O0', >>>> '--download-kokkos', >>>> '--download-kokkos-kernels', >>>> '--download-kokkos-commit=3.4.01', >>>> '--download-kokkos-kernels-commit=3.4.01', >>>> '--with-kokkos-hip-arch=VEGA908', >>>> >>>> and 'make check' ran smoothly on a compute node >>>> >>>> $ make check >>>> Running check examples to verify correct installation >>>> Using PETSC_DIR=/ccs/home/jczhang/petsc and >>>> PETSC_ARCH=arch-spock-cray-kokkos-dbg >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI >>>> process >>>> C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI >>>> processes >>>> C/C++ example src/snes/tutorials/ex3k run successfully with >>>> kokkos-kernels >>>> Fortran example src/snes/tutorials/ex5f run successfully with 1 MPI >>>> process >>>> >>>> Completed test examples >>>> >>>> >>>> --Junchao Zhang >>>> >>>> >>>> On Fri, Jul 16, 2021 at 6:04 PM Mark Adams <mfad...@lbl.gov> wrote: >>>> >>>>> And I find that this error, on non-Kokkos C tests, is fixed by turning >>>>> the fortran bindings off: >>>>> >>>>> ld.lld: error: >>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_start.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_size.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> ld.lld: error: >>>>> /gpfs/alpine/phy122/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib/libpetsc.so: >>>>> undefined reference to .omp_offloading.img_cache.cray_amdgcn-amd-amdhsa >>>>> [--no-allow-shlib-undefined] >>>>> >>>>> On Fri, Jul 16, 2021 at 3:53 PM Mark Adams <mfad...@lbl.gov> wrote: >>>>> >>>>>> Not complex. THis has some overlap with my problem w/o Kokkos. >>>>>> >>>>>> On Fri, Jul 16, 2021 at 12:54 PM Junchao Zhang < >>>>>> junchao.zh...@gmail.com> wrote: >>>>>> >>>>>>> Do you use complex? post your configure.log. >>>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 16, 2021 at 9:47 AM Mark Adams <mfad...@lbl.gov> wrote: >>>>>>> >>>>>>>> The simple Kokkos example is failing for me on Spock. >>>>>>>> Any ideas? >>>>>>>> Thanks, >>>>>>>> >>>>>>>> 10:44 main *= >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials$ make >>>>>>>> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >>>>>>>> PETSC_ARCH=arch-spock-opt-cray-kokkos ex3k >>>>>>>> MPICH_CXX="hipcc" OMPI_CXX="hipcc" CC >>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lhsa-runtime64 >>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >>>>>>>> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >>>>>>>> -fstack-protector -fvisibility=hidden -g -O2 -fPIC -Wall >>>>>>>> -Wwrite-strings >>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fvisibility=hidden -g -O3 >>>>>>>> -std=c++14 -I/gpfs/alpine/csc314/scratch/adams/petsc/include >>>>>>>> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/include >>>>>>>> -I/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include >>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >>>>>>>> -I/sw/spock/spack-envs/views/rocm-4.1.0/include ex3k.kokkos.cxx >>>>>>>> >>>>>>>> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-opt-cray-kokkos/lib >>>>>>>> -Wl,-rpath,/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>> -L/gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/lib >>>>>>>> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >>>>>>>> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>> -L/opt/cray/pe/mpich/8.1.4/gtl/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >>>>>>>> -L/opt/cray/pe/pmi/6.0.10/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >>>>>>>> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >>>>>>>> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >>>>>>>> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >>>>>>>> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>>>>>>> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >>>>>>>> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>>>>>>> -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu >>>>>>>> -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread >>>>>>>> -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 >>>>>>>> -lquadmath -lstdc++ -ldl -o ex3k >>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:259:11: >>>>>>>> error: no member named 'lock_address_host_space' in namespace >>>>>>>> 'Kokkos::Impl::Kokkos::Impl'; did you mean simply >>>>>>>> 'lock_address_host_space'? >>>>>>>> while (!Impl::lock_address_host_space((void*)dest)) >>>>>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>>>>>> lock_address_host_space >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_HostSpace.hpp:83:6: >>>>>>>> note: 'lock_address_host_space' declared here >>>>>>>> bool lock_address_host_space(void* ptr); >>>>>>>> ^ >>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:261:3: >>>>>>>> error: no member named 'memory_fence' in namespace >>>>>>>> 'Kokkos::Impl::Kokkos'; >>>>>>>> did you mean simply 'memory_fence'? >>>>>>>> Kokkos::memory_fence(); >>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>> memory_fence >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>> note: 'memory_fence' declared here >>>>>>>> void memory_fence() { >>>>>>>> ^ >>>>>>>> In file included from ex3k.kokkos.cxx:3: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscdmda_kokkos.hpp:4: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec_kokkos.hpp:14: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscvec.h:9: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsys.h:42: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/csc314/scratch/adams/petsc/include/petscsystypes.h:255: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Complex.hpp:47: >>>>>>>> In file included from >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/Kokkos_Atomic.hpp:212: >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Atomic_Exchange.hpp:277:3: >>>>>>>> error: no member named 'memory_fence' in namespace >>>>>>>> 'Kokkos::Impl::Kokkos'; >>>>>>>> did you mean simply 'memory_fence'? >>>>>>>> Kokkos::memory_fence(); >>>>>>>> ^~~~~~~~~~~~~~~~~~~~ >>>>>>>> memory_fence >>>>>>>> /gpfs/alpine/geo127/proj-shared/spock/petsc/current/arch-opt-cray-kokkos/include/impl/Kokkos_Memory_Fence.hpp:53:6: >>>>>>>> note: 'memory_fence' declared here >>>>>>>> void memory_fence() { >>>>>>>> ^ >>>>>>>> >>>>>>>