Here is a summary of the installation process on Cray XC50. I have configured deal.II with MPI, LAPACK, SCALAPACK, PETSc and p4est. Our system didn't have p4est so I started with installing it. All cray libraries are in /opt/cray/pe/lib64/ in out system.
*Installing p4est* 1. Download source files and setup script from here <https://www.dealii.org/current/external-libs/p4est.html>. 2. By default, the setup script searches for mpicxx compilers. Instead, explicitly specifiy cray compilers. The configure command will look as follows. "$SRCDIR/configure" CXX=/opt/cray/pe/craype/2.5.13/bin/CC \ CC=/opt/cray/pe/craype/2.5.13/bin/cc \ F77=/opt/cray/pe/craype/2.5.13/bin/ftn \ FC=/opt/cray/pe/craype/2.5.13/bin/ftn \ --enable-mpi --enable-shared \ --disable-vtk-binary --without-blas \ --prefix="$INSTALL_DEBUG" CFLAGS="$CFLAGS_DEBUG" \ CPPFLAGS="-DSC_LOG_PRIORITY=SC_LP_ESSENTIAL" \ "$@" > config.output || bdie "Error in configure" Make this change for both FAST and DEBUG versions. 3. By default cray assumes static linking. Change them: export XTPE_LINK_TYPE=dynamic export CRAYPE_LINK_TYPE=dynamic This will be required in subsequent steps also. 4. The makefile generated uses flags corresponding to GNU compilers. Switch module PrgEnv-cray with PrgEnv-gnu. *Configuring with LAPACK and SCALAPACK* 1. For LAPACK, deal.II's find module calls cmake's corresponding find module. For this to work on cray systems, cmake version >=3.16 is required. So I installed a new version in my home directory and used this cmake version. See this <https://stackoverflow.com/questions/54681204/cray-cc-wrapper-cmake-find-package-blas> and this <https://gitlab.kitware.com/cmake/cmake/merge_requests/3451>. 2. For Cray environments, lapack libraries are linked directly to cray compiler without requiring any other flags. So the _lapack_libraries variable in deal.II's FindLAPACK.cmake will be empty. This is okay. So set this as OPTIONAL in the end of this file. 3. For SCALAPACK, the library name in FindLAPACK.cmake should be changed to sci_gnu_61_mpi_mp (or whatever is the name of libsci library on your system) since on cray, SCALAPACK is a part of this library. *Configuring with MPI and PETSc* 1. For MPI, simply specify the compilers explicitly. 2. For PETSc, the library name must be changed to craypetsc_gnu_real-64 (depending on your system). 3. The additional libraries PETSc interfaces to are read from linker line of $PETSC_DIR/lib/petsc/conf/petscvariables. Make a copy of this file and modify the linker line so that the library names are correct (if they are not already, as was the case with me). Change the hint to petscvariables file in FindPETSC.cmake. 4. Also, add the correct hint to these library paths in the following portion of the aforementioned file. DEAL_II_FIND_LIBRARY(PETSC_LIBRARY_${_token} NAMES ${_token} #HINTS ${_hints} HINTS ${_hints} ${CMAKE_PREFIX_PATH} ) In my case, I set CMAKE_PREFIX_PATH in configure script to /opt/cray/pe/lib64. 5. If your system has PETSc libraries with ".so.mpi<xxx>" extensions, you must enable find those in dealii-9.1.1/CMakeLists.txt (the top most one) SET(CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES} ".so.0" ".so.5" ".so.mpi31.2" ".so.mpi31.4" ".so.mpi31.5" ".so.mpi31.6" ".so.mpi31.12" ) 6. If you are using 64-bit versions of PETSc libraries, you must enable this for deal.II too (see below). You must unload the atp module before configuring (see here <https://groups.google.com/forum/?hl=en-GB#!topic/dealii/9IMUhsjSGZ8>). For cross-compilation (see here <https://groups.google.com/forum/?hl=en-GB#!searchin/dealii/cross$20compilation%7Csort:date/dealii/k4ZU250SAu8/R5AJBe91BgAJ>), you can just add -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment without requiring a Toolchain file in newer cmake versions. The configure script is cmake_new=~/bin/cmake-3.16.4/usr/local/bin/cmake # from bashrc, shell scripts can't use aliases $cmake_new -DCMAKE_INSTALL_PREFIX=~/bin/dealii-9.1.1/ \ -DWITH_64BIT_INDICES=ON \ -DCMAKE_PREFIX_PATH=/opt/cray/pe/lib64 \ -DWITH_MPI=ON \ -DMPI_DIR=/opt/cray/pe/mpt/default/gni/mpich-gnu/5.1/ \ -DMPI_CXX_INCLUDE_PATH=/opt/cray/pe/mpt/default/gni/mpich-gnu/5.1/include/ \ -DCMAKE_CXX_COMPILER=/opt/cray/pe/craype/2.5.13/bin/CC \ -DCMAKE_C_COMPILER=/opt/cray/pe/craype/2.5.13/bin/cc \ -DCMAKE_Fortran_COMPILER=/opt/cray/pe/craype/2.5.13/bin/ftn \ -DWITH_BLAS=ON \ -DWITH_LAPACK=ON \ -DWITH_SCALAPACK=ON \ -DWITH_PETSC=ON \ -DWITH_P4EST=ON -DP4EST_DIR=~/bin/p4est-2.2/ \ -DCMAKE_SYSTEM_NAME=CrayLinuxEnvironment \ ~/source/dealii-9.1.1 -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/63bfc926-077d-45d3-85d8-fa6793c1baf1%40googlegroups.com.
