https://gcc.gnu.org/bugzilla/show_bug.cgi?id=127684
Bug ID: 127684
Summary: caf_shmem: SYNC IMAGES hangs when an image in the
image set stops or fails
Product: gcc
Version: 17.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: libfortran
Assignee: unassigned at gcc dot gnu.org
Reporter: jvdelisle at gcc dot gnu.org
Target Milestone: ---
This is a new bug found while testing for PR127437. This was found using Claude
to run variation tests under heavy loads. Getting a deterministic reproducer
has been difficult at best.
With -fcoarray=shared, SYNC IMAGES (image-list, STAT=) hangs when an
image of the image set initiates normal termination (STOP) or fails
(FAIL IMAGE) while another image of the set is already waiting in the
statement. Whether it hangs depends on the order in which the images
arrive, so it shows up intermittently under load; the reproducers
below force the order and hang every time on trunk (f4d32f612a3):
$ gfortran -fcoarray=shared sync_images_failed_1.f90
$ GFORTRAN_NUM_IMAGES=3 ./a.out # hangs, 3 of 3 runs
---- sync_images_failed_1.f90 ----
program sync_images_failed_1
use iso_fortran_env, only : atomic_int_kind, stat_failed_image
implicit none
integer(atomic_int_kind) :: arrived[*], a
integer :: st, x[*]
if (num_images () < 3) stop
arrived = 0
x = 0
sync all
select case (this_image ())
case (1)
call atomic_define (arrived[3], 1)
st = 0
sync images ([2, 3], stat=st)
if (st /= stat_failed_image) stop 1
if (x[2] /= 42) stop 2
case (2)
do while (image_status (3) /= stat_failed_image)
end do
x = 42
st = 0
sync images ([1, 3], stat=st)
if (st /= stat_failed_image) stop 3
case (3)
a = 0
do while (a == 0)
call atomic_ref (a, arrived)
end do
call spin ()
fail image
end select
contains
subroutine spin ()
integer :: c
integer(kind=8) :: v
v = 2
do c = 1, 20000000
v = mod (v * 2, 199679_8)
end do
end subroutine
end program sync_images_failed_1
----
Replacing FAIL IMAGE by STOP (and stat_failed_image by
stat_stopped_image) hangs the same way.
Image 1 enters SYNC IMAGES while image 3 is still active; image 2
enters only after image 3 has terminated. Two defects combine:
1. _gfortran_caf_sync_images returns early with STAT_FAILED_IMAGE or
STAT_STOPPED_IMAGE, without synchronizing, when an image of the
list has already terminated. Image 1 counted its synchronization
with image 2 before image 3 terminated and keeps waiting for
image 2, which never synchronizes with it.
F2023 11.7.11 p5/p6: with a failed image involved, "the intended
action is performed on the active images involved", so image 2
has to synchronize with image 1. With a stopped image involved
the statement has the effect of SYNC MEMORY, so image 1 has to stop
waiting -- without leaving its synchronization with image 2
counted, or the next SYNC IMAGES of images 1 and 2 pairs wrongly.
2. The supervisor wakes images waiting in SYNC IMAGES after it reaped
a terminated image (supervisor.c), but signals the condition
variables without holding sync_images_table_lock. An image that
checked the status of its partners and is preempted before it
waits misses that wakeup and sleeps forever.
SYNC IMAGES (*) does not have the early return of 1. and was not seen
to hang. Found with a fuzz matrix of terminating images; under
16-way CPU load the image-list cases hang in 6 of 10 runs.
