Hi Jerry,
good you could build Toon's code.
Your idea of using the OpenCoarray tests to test caf_shmem made me think about
how to do it the easiest.
I came up with the following:
1. Pull a recent OpenCoarray source tree from Github or use a clean existing
one.
2. Apply attached patch.
3. Create a build directory for OpenCoarrays.
4. Copy or link libcaf_shmem.a from a recent gfortran build to the OpenCoarray
build directory.
5. Do a cmake <OpenCoarraysSrcDir> in the build directory.
6. Now cmake --build . && ctest . should test using caf_shmem.
Notes:
- Some tests have been deactivated, because they use MPI-features, that are not
present in caf_shmem.
- There are still two test failing about some test on cafrun. Those are ok,
because the file has been patched and I can't find where the test is coming
from. Unimportant!
- You need an update patch set for caf_shmem, because the OpenCoarray tests
revealed some/a lot of issues in caf_shmem. I am currently regtesting the
patch and will publish as soon as it regtests cleanly.
Regards,
Andre
On Fri, 4 Jul 2025 10:51:43 -0700
Jerry D <[email protected]> wrote:
> On 7/4/25 5:12 AM, Andre Vehreschild wrote:
> > Hi all,
> >
> > attached patches goes on top of other 6 caf_shmem coarray patches and fixes
> > missing includes esp. on non-Linux systems. I have tested this on a FreeBSD,
> > which is very time consuming due to it being fully virtualized on my system.
> >
> > Regtests ok on x86_64-pc-linux-gnu and aarch64-unknown-freebsd14.3. Ok for
> > mainline?
> >
> > Thanks to Steve for bringing these deficiencies to my attention.
> >
> > Regards,
> > Andre
>
> So far,
>
> $ export GFORTRAN_NUM_IMAGES=9
> $ rm *.mod
> $ gfc -fcoarray=lib random-weather.f90 -lcaf_shmem
> $ ./a.out
> Decomposition information on image 6 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 1 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 4 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 5 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 9 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 3 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 8 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 2 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> Decomposition information on image 7 : there are 9 * 1 slabs; the slabs
> are 8 * 70 grid cells in size.
> .
> .
> .
> Time 3600 Image 4 PS= 99925.0391 T= 301.282928
> U= -51.2542686 V= 24.3605309 W= -0.296301365 Q=
> 1.48258626E-03
> Time 3600 Image 9 PS= 99899.3047 T= 299.897095
> U= 62.8683090 V= -57.9342270 W= 0.445489585 Q=
> 1.90666097E-03
> Time 3600 Image 1 PS= 99966.7656 T= 300.011597
> U= -1.93229961 V= -118.892410 W= -6.45965934E-02 Q=
> 2.03774264E-03
> Time 3600 Image 7 PS= 100015.938 T= 300.066162
> U= -17.6038494 V= -0.982973158 W= 7.21789524E-02 Q=
> 2.17592530E-03
> Time 3600 Image 2 PS= 100003.477 T= 300.078522
> U= -2.38964367 V= -18.8026981 W= -0.179861650 Q=
> 1.99834118E-03
> Time 3600 Image 5 PS= 100077.422 T= 300.781494
> U= -16.6273994 V= -101.607895 W= 0.361649722 Q=
> 1.74388883E-03
> Time 3600 Image 3 PS= 100002.391 T= 299.708862
> U= 18.6304798 V= 0.391739845 W= 2.24014949E-02 Q=
> 1.96914421E-03
> Time 3600 Image 8 PS= 100074.359 T= 299.516235
> U= -55.1445618 V= 68.3090286 W= -0.537869334 Q=
> 2.32057413E-03
> Time 3600 Image 6 PS= 99976.4453 T= 300.221924
> U= -1.62557888 V= 1.44226456 W= 0.201509774 Q=
> 1.97460176E-03
> $
>
>
> real 0m0.066s
> user 0m0.337s
> sys 0m0.107s
>
> Definitely much faster than mpich. I also over prescribed the number of
> images to 30 and ran as well.
>
> I still need to build OpenCoarrays using this gfortran-16 and make sure it
> succeeds those tests with mpich. I will try to then test each case on the
> OpenCoarrays suite of tests with -lcaf_shmem and see if those all work.
>
> Any ideas on how to stress test this. I only have 32 gig of memory here and
> would like to see how a longer running program does.
>
> Regards,
>
> Jerry
--
Andre Vehreschild * Email: vehre ad gmx dot de
From fad1373c34fe88dccb4417f408f280dd4113e7c0 Mon Sep 17 00:00:00 2001
From: Andre Vehreschild <[email protected]>
Date: Wed, 9 Jul 2025 16:14:59 +0200
Subject: [PATCH] Use caf_shmem for build an test.
---
CMakeLists.txt | 30 ++++++++++++++-------
src/script-templates/caf.in | 2 +-
src/script-templates/cafrun.in | 7 ++++-
src/tests/unit/init_register/CMakeLists.txt | 2 +-
src/tests/unit/teams/CMakeLists.txt | 2 +-
5 files changed, 29 insertions(+), 14 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index e9694b4..657e057 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -771,6 +771,7 @@ function(add_caf_test name num_caf_img test_target)
add_test(NAME ${name} COMMAND "${BASH_EXECUTABLE}" "${CMAKE_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}/cafrun" ${test_parameters} "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${test_target}")
endif()
set_property(TEST ${name} PROPERTY PASS_REGULAR_EXPRESSION "Test passed.")
+ set_property(TEST ${name} PROPERTY TIMEOUT 5)
endfunction(add_caf_test)
#--------------
@@ -782,7 +783,7 @@ if(opencoarrays_aware_compiler)
message ( STATUS "Running Developer tests is enabled. Some tests may fail for open issues." )
endif()
# Unit tests targeting each libcaf_mpi function, argument, and branch of code
- add_caf_test(initialize_mpi 2 initialize_mpi)
+ #add_caf_test(initialize_mpi 2 initialize_mpi)
add_caf_test(register 2 register)
add_caf_test(register_vector 2 register_vector)
add_caf_test(register_alloc_vector 2 register_alloc_vector)
@@ -808,7 +809,7 @@ if(opencoarrays_aware_compiler)
if(NOT CMAKE_Fortran_COMPILER_VERSION VERSION_LESS 8)
add_caf_test(team_number 8 team_number)
add_caf_test(teams_subset 3 teams_subset)
- add_caf_test(get_communicator 3 get_communicator)
+ #add_caf_test(get_communicator 3 get_communicator)
add_caf_test(teams_coarray_get 5 teams_coarray_get)
add_caf_test(teams_coarray_get_by_ref 5 teams_coarray_get_by_ref)
add_caf_test(teams_coarray_send 5 teams_coarray_send)
@@ -816,7 +817,7 @@ if(opencoarrays_aware_compiler)
add_caf_test(teams_coarray_sendget 5 teams_coarray_sendget)
add_caf_test(sync_team 8 sync_team)
add_caf_test(alloc_comp_multidim_shape 2 alloc_comp_multidim_shape)
- set_tests_properties(alloc_comp_multidim_shape PROPERTIES TIMEOUT 300)
+ set_tests_properties(alloc_comp_multidim_shape PROPERTIES TIMEOUT 20)
endif()
if(NOT CMAKE_Fortran_COMPILER_VERSION VERSION_LESS 16)
add_caf_test(teams_this_image 8 teams_this_image)
@@ -829,8 +830,11 @@ if(opencoarrays_aware_compiler)
if (gfortran_compiler)
if((NOT CMAKE_Fortran_COMPILER_VERSION VERSION_LESS 7.0.0) OR (CAF_RUN_DEVELOPER_TESTS OR $ENV{OPENCOARRAYS_DEVELOPER}))
add_caf_test(send_convert_nums 2 send_convert_nums)
+ set_tests_properties(send_convert_nums PROPERTIES TIMEOUT 20)
add_caf_test(sendget_convert_nums 3 sendget_convert_nums)
+ set_tests_properties(sendget_convert_nums PROPERTIES TIMEOUT 20)
add_caf_test(sendget_convert_char_array 3 sendget_convert_char_array)
+ set_tests_properties(sendget_convert_char_array PROPERTIES TIMEOUT 20)
else()
message( AUTHOR_WARNING "Skipping the following tests due to GFortran < 7.0.0 lack of compatibility:
sendget_convert_nums.f90
@@ -839,7 +843,9 @@ if(opencoarrays_aware_compiler)
endif()
if((NOT CMAKE_Fortran_COMPILER_VERSION VERSION_LESS 7.3.0) OR (CAF_RUN_DEVELOPER_TESTS OR $ENV{OPENCOARRAYS_DEVELOPER}))
add_caf_test(send_convert_char_array 2 send_convert_char_array)
+ set_tests_properties(send_convert_char_array PROPERTIES TIMEOUT 20)
add_caf_test(alloc_comp_send_convert_nums 2 alloc_comp_send_convert_nums)
+ set_tests_properties(alloc_comp_send_convert_nums PROPERTIES TIMEOUT 20)
elseif((NOT CMAKE_Fortran_COMPILER_VERSION VERSION_LESS 7.4.0) OR (CAF_RUN_DEVELOPER_TESTS OR $ENV{OPENCOARRAYS_DEVELOPER}))
add_caf_test(send-strided-self 2 send-strided-self)
endif()
@@ -859,6 +865,7 @@ if(opencoarrays_aware_compiler)
# Pure get tests
add_caf_test(get_array 2 get_array)
+ set_tests_properties(get_array PROPERTIES TIMEOUT 20)
add_caf_test(get_self 2 get_self)
add_caf_test(get_convert_nums 2 get_convert_nums)
add_caf_test(get_convert_char_array 2 get_convert_char_array)
@@ -869,8 +876,10 @@ if(opencoarrays_aware_compiler)
# Pure send tests
add_caf_test(send_array 2 send_array)
+ set_tests_properties(send_array PROPERTIES TIMEOUT 20)
add_caf_test(convert-before-put 3 convert-before-put)
add_caf_test(send_with_vector_index 2 send_with_vector_index)
+ set_tests_properties(send_with_vector_index PROPERTIES TIMEOUT 20)
# Pure sendget tests
add_caf_test(strided_sendget 3 strided_sendget)
@@ -883,6 +892,7 @@ if(opencoarrays_aware_compiler)
add_caf_test(co_broadcast_derived_type 4 co_broadcast_derived_type_test)
if((gfortran_compiler AND (NOT CMAKE_Fortran_COMPILER_VERSION VERSION_LESS 10.0.0)) OR (CAF_RUN_DEVELOPER_TESTS OR $ENV{OPENCOARRAYS_DEVELOPER}))
add_caf_test(co_broadcast_allocatable_components 4 co_broadcast_allocatable_components_test)
+ set_tests_properties(co_broadcast_allocatable_components PROPERTIES TIMEOUT 20)
endif()
if((gfortran_compiler AND (NOT CMAKE_Fortran_COMPILER_VERSION VERSION_LESS 11.2.2)) OR (CAF_RUN_DEVELOPER_TESTS OR $ENV{OPENCOARRAYS_DEVELOPER}))
add_caf_test(co_broadcast_alloc_mixed 2 co_broadcast_alloc_mixed)
@@ -1007,8 +1017,8 @@ else()
endif()
# Test bash installation scripts
-include(cmake/AddInstallationScriptTest.cmake )
-add_installation_script_test(installation-scripts.sh src/tests/installation/)
+#nclude(make/AddInstallationScriptTest.cmake )
+#add_installation_script_test(installation-scripts.sh src/tests/installation/)
if( CAF_ENABLE_FAILED_IMAGES AND ( NOT
( "${CMAKE_C_COMPILER_ID}" STREQUAL "GNU" ) AND ( "${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "x86_64" ) )
@@ -1021,11 +1031,11 @@ set(TRAVIS_SCRIPTS
install.osx.sh
test-script.InstallScript.sh
test-script.cmake.sh)
-foreach(SCRIPT ${TRAVIS_SCRIPTS})
- if(EXISTS "${CMAKE_SOURCE_DIR}/developer-scripts/travis/${SCRIPT}")
- lint_script("${CMAKE_SOURCE_DIR}/developer-scripts/travis" ${SCRIPT})
- endif()
-endforeach()
+#foreach(SCRIPT ${TRAVIS_SCRIPTS})
+# if(EXISTS "${CMAKE_SOURCE_DIR}/developer-scripts/travis/${SCRIPT}")
+# lint_script("${CMAKE_SOURCE_DIR}/developer-scripts/travis" ${SCRIPT})
+# endif()
+#endforeach()
# Create and install a manifest of installed files, and another containing their sha256 hashes
foreach(VAR CMAKE_BINARY_DIR CMAKE_INSTALL_DATADIR PROJECT_NAME SOURCE_DATE_EPOCH)
diff --git a/src/script-templates/caf.in b/src/script-templates/caf.in
index ab0e5f8..30762ab 100755
--- a/src/script-templates/caf.in
+++ b/src/script-templates/caf.in
@@ -86,7 +86,7 @@ mpi_link_flags=(@CAF_MPI_Fortran_LINK_FLAGS@) # e.g. `pkg-config
# __*AND*__ `pkg-config
# --libs-only-L`
mpi_compile_flags=(@CAF_MPI_Fortran_COMPILE_FLAGS@)
-caf_libs=(@CAF_LIBS@) # e.g. "libcaf_mpi" "libcaf_extensions",
+caf_libs=(libcaf_shmem.a) # e.g. "libcaf_mpi" "libcaf_extensions",
threads_lib=@THREADS_LIB@ # pthreads or compatible, needed for windows
# preferably full paths, but could be
# combination of -L... and -lcaf_mpi...
diff --git a/src/script-templates/cafrun.in b/src/script-templates/cafrun.in
index dff2f1f..b365925 100755
--- a/src/script-templates/cafrun.in
+++ b/src/script-templates/cafrun.in
@@ -46,6 +46,11 @@ set -o pipefail
# Turn on traces, useful while debugging but commented out by default
# set -o xtrace
+shift 1
+export GFORTRAN_NUM_IMAGES=$1
+shift 1
+exec $*
+exit $?
#---------------------
# Configured variables
@@ -268,7 +273,7 @@ elif [[ "${1}" == -np || "${1}" == -n ]]; then
mpiexec_args+=("${@:4:$((${#} - 3))}")
fi
- if "${CAFRUN}" "${mpiexec_args[@]//''/}" ; then
+ if "${*}" ; then
exit $?
else
return_code=$?
diff --git a/src/tests/unit/init_register/CMakeLists.txt b/src/tests/unit/init_register/CMakeLists.txt
index a7d29d0..66368de 100644
--- a/src/tests/unit/init_register/CMakeLists.txt
+++ b/src/tests/unit/init_register/CMakeLists.txt
@@ -1,4 +1,4 @@
-caf_compile_executable(initialize_mpi initialize_mpi.F90)
+#caf_compile_executable(initialize_mpi initialize_mpi.F90)
caf_compile_executable(register register.f90)
set_target_properties(build_register
PROPERTIES MIN_IMAGES 2)
diff --git a/src/tests/unit/teams/CMakeLists.txt b/src/tests/unit/teams/CMakeLists.txt
index 2ef2ee8..825401e 100644
--- a/src/tests/unit/teams/CMakeLists.txt
+++ b/src/tests/unit/teams/CMakeLists.txt
@@ -1,6 +1,6 @@
caf_compile_executable(team_number team_number.F90)
caf_compile_executable(teams_subset teams_subset.f90)
-caf_compile_executable(get_communicator get-communicator.F90)
+#caf_compile_executable(get_communicator get-communicator.F90)
caf_compile_executable(teams_coarray_get teams_coarray_get.f90)
caf_compile_executable(teams_coarray_get_by_ref teams_coarray_get.f90)
caf_compile_executable(teams_coarray_send teams_coarray_send.f90)
--
2.50.0
