Re: [petsc-dev] Kokkos make error on Spock
Mark, I can reproduce this error with PrgEnv-cray, i.e., using the Cray compiler (clang-11). Previously I used PrgEnv-gnu, which did not have this error. Probably it is a problem of Spock. But I am not sure. --Junchao Zhang On Sat, Jul 17, 2021 at 10:17 AM Mark Adams wrote: > And I can run a fortran test, with warnings, but C tests fail: > > 11:15 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > PETSC_ARCH=arch-spock-opt-cray-kokkos > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > FC > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > FLINKER > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > TEST > arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > > > On Sat, Jul 17, 2021 at 10:53 AM Mark Adams wrote: > >> HUmm, I can not reproduce this. >> >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-dbg-kokkos check >> Running check examples to verify correct installation >> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >> PETSC_ARCH=arch-spock-dbg-kokkos >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex19.PETSc] Error 2 (ignored) >> ***Error detected during compile or >> link!*** >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >> >> * >> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex19.c >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >>
Re: [petsc-dev] Kokkos make error on Spock
Also, the application is a C++ code with PETSc Fortran methods. Thanks, Mark On Sat, Jul 17, 2021 at 11:17 AM Mark Adams wrote: > And I can run a fortran test, with warnings, but C tests fail: > > 11:15 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test > search='ts_utils_dmplexlandau_tutorials-ex1f90_0' > Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 > PETSC_ARCH=arch-spock-opt-cray-kokkos > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > FC > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > FLINKER > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 128 of symbol > `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: > warning: alignment 64 of symbol `$data_init$iso_c_binding_' in > /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in > arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o > TEST > arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts > ok ts_utils_dmplexlandau_tutorials-ex1f90_0 > ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 > > > On Sat, Jul 17, 2021 at 10:53 AM Mark Adams wrote: > >> HUmm, I can not reproduce this. >> >> 10:49 jczhang/fix-cray-mpicxx-includes/main= >> /gpfs/alpine/csc314/scratch/adams/petsc$ make >> PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc >> PETSC_ARCH=arch-spock-dbg-kokkos check >> Running check examples to verify correct installation >> Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and >> PETSC_ARCH=arch-spock-dbg-kokkos >> gmake[3]: >> [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: >> ex19.PETSc] Error 2 (ignored) >> ***Error detected during compile or >> link!*** >> See http://www.mcs.anl.gov/petsc/documentation/faq.html >> /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 >> >> * >> cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 >> -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC >> -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas >> -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/include >> -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/include >> -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex19.c >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib >> -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -L/sw/spock/spack-envs/views/rocm-4.1.0/lib >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 >> -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib >> -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib >> -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib >> -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib >> -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux >> -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 >> -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib >> -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse >> -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 >> -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray >>
Re: [petsc-dev] Kokkos make error on Spock
And I can run a fortran test, with warnings, but C tests fail: 11:15 jczhang/fix-cray-mpicxx-includes/main= /gpfs/alpine/csc314/scratch/adams/petsc$ make PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc PETSC_ARCH=arch-spock-opt-cray-kokkos -f gmakefile test search='ts_utils_dmplexlandau_tutorials-ex1f90_0' Using MAKEFLAGS: -- search=ts_utils_dmplexlandau_tutorials-ex1f90_0 PETSC_ARCH=arch-spock-opt-cray-kokkos PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc FC arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o FLINKER arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90 /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 128 of symbol `$host_init$$runtime_init_for_iso_c_binding$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o /opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: warning: alignment 64 of symbol `$data_init$iso_c_binding_' in /opt/cray/pe/cce/11.0.4/cce/x86_64/lib/libmodules.so is smaller than 256 in arch-spock-opt-cray-kokkos/tests/ts/utils/dmplexlandau/tutorials/ex1f90.o TEST arch-spock-opt-cray-kokkos/tests/counts/ts_utils_dmplexlandau_tutorials-ex1f90_0.counts ok ts_utils_dmplexlandau_tutorials-ex1f90_0 ok diff-ts_utils_dmplexlandau_tutorials-ex1f90_0 On Sat, Jul 17, 2021 at 10:53 AM Mark Adams wrote: > HUmm, I can not reproduce this. > > 10:49 jczhang/fix-cray-mpicxx-includes/main= > /gpfs/alpine/csc314/scratch/adams/petsc$ make > PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc > PETSC_ARCH=arch-spock-dbg-kokkos check > Running check examples to verify correct installation > Using PETSC_DIR=/gpfs/alpine/csc314/scratch/adams/petsc and > PETSC_ARCH=arch-spock-dbg-kokkos > gmake[3]: > [/gpfs/alpine/csc314/scratch/adams/petsc/lib/petsc/conf/rules:301: > ex19.PETSc] Error 2 (ignored) > ***Error detected during compile or > link!*** > See http://www.mcs.anl.gov/petsc/documentation/faq.html > /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/tutorials ex19 > > * > cc -L/sw/spock/spack-envs/views/rocm-4.1.0/lib -lamdhip64 -lhsa-runtime64 > -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 -fPIC > -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas > -fstack-protector -Qunused-arguments -fvisibility=hidden -g -O0 > -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/include > -I/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/include > -I/sw/spock/spack-envs/views/rocm-4.1.0/includeex19.c > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -L/gpfs/alpine/csc314/scratch/adams/petsc/arch-spock-dbg-kokkos/lib > -Wl,-rpath,/sw/spock/spack-envs/views/rocm-4.1.0/lib > -L/sw/spock/spack-envs/views/rocm-4.1.0/lib > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib64 -L/opt/gcc/8.1.0/snos/lib64 > -Wl,-rpath,/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -L/opt/cray/pe/libsci/21.04.1.1/CRAY/9.0/x86_64/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -L/opt/cray/pe/mpich/8.1.4/ofi/cray/9.1/lib > -Wl,-rpath,/opt/cray/pe/mpich/8.1.4/gtl/lib > -L/opt/cray/pe/mpich/8.1.4/gtl/lib -Wl,-rpath,/opt/cray/pe/pmi/6.0.10/lib > -L/opt/cray/pe/pmi/6.0.10/lib -Wl,-rpath,/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -L/opt/cray/pe/dsmml/0.1.4/dsmml/lib > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -L/opt/cray/pe/cce/11.0.4/cce/x86_64/lib > -Wl,-rpath,/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -L/opt/cray/xpmem/2.2.40-2.1_2.7__g3cf3325.shasta/lib64 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -L/opt/cray/pe/cce/11.0.4/cce-clang/x86_64/lib/clang/11.0.0/lib/linux > -Wl,-rpath,/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -L/opt/gcc/8.1.0/snos/lib/gcc/x86_64-suse-linux/8.1.0 > -Wl,-rpath,/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -L/opt/cray/pe/cce/11.0.4/binutils/x86_64/x86_64-pc-linux-gnu/..//x86_64-unknown-linux-gnu/lib > -lpetsc -lkokkoskernels -lkokkoscontainers -lkokkoscore -lhipsparse > -lhipblas -lrocsparse -lrocsolver -lrocblas -lamdhip64 -lhsa-runtime64 > -lstdc++ -ldl -lpmi -lsci_cray_mpi -lsci_cray -lmpifort_cray -lmpi_cray > -lmpi_gtl_hsa -lxpmem -ldsmml -lpgas-shmem -lquadmath -lcrayacc_amdgpu > -lopenacc -lmodules -lfi -lcraymath -lf -lu -lcsup -lgfortran -lpthread > -lgcc_eh -lm -lclang_rt.craypgo-x86_64 -lclang_rt.builtins-x86_64 > -lquadmath -lstdc++ -ldl -o ex19 >
