[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #14 from GCC Commits --- The releases/gcc-15 branch has been updated by Tobias Burnus : https://gcc.gnu.org/g:c38760cdb2ac21f690647e965acc0eaba3250f6d commit r15-9740-gc38760cdb2ac21f690647e965acc0eaba3250f6d Author: Tobias Burnus Date: Wed May 28 15:14:14 2025 +0200 libgomp.fortran/metadirective-1.f90: Expect 'error:' for nvptx compile [PR118694] This should have been part of commit r16-838-gb3d07ec7ac2ccd or r16-883-g5d6ed6d604ff94 - all showing the same issue: '!$omp target' followed by a metadirective with 'teams'; if the metadirective cannot be early resolved, a diagnostic error is shown about using directives between 'target' and 'teams'. While the message is misleading, the problem is that the host invokes 'target' differently when 'teams' is present; in this case, host fallback + amdgcn offload require the no-teams case, nvptx offload the teams case such that it only can be resolved at runtime. Mark the error as 'dg-bogus + xfail' to silence the FAIL, when nvptx offloading is compiled for. (If not, the metadirective can be resolved early during compilation.) libgomp/ChangeLog: PR middle-end/118694 * testsuite/libgomp.fortran/metadirective-1.f90: xfail when compiling (also) for nvptx offloading as an error is then expected. (cherry picked from commit 5b2e7afb135aa10e17e51b891d4b2c85ee541ade)
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #13 from GCC Commits --- The master branch has been updated by Tobias Burnus : https://gcc.gnu.org/g:5b2e7afb135aa10e17e51b891d4b2c85ee541ade commit r16-935-g5b2e7afb135aa10e17e51b891d4b2c85ee541ade Author: Tobias Burnus Date: Wed May 28 15:14:14 2025 +0200 libgomp.fortran/metadirective-1.f90: Expect 'error:' for nvptx compile [PR118694] This should have been part of commit r16-838-gb3d07ec7ac2ccd or r16-883-g5d6ed6d604ff94 - all showing the same issue: '!$omp target' followed by a metadirective with 'teams'; if the metadirective cannot be early resolved, a diagnostic error is shown about using directives between 'target' and 'teams'. While the message is misleading, the problem is that the host invokes 'target' differently when 'teams' is present; in this case, host fallback + amdgcn offload require the no-teams case, nvptx offload the teams case such that it only can be resolved at runtime. Mark the error as 'dg-bogus + xfail' to silence the FAIL, when nvptx offloading is compiled for. (If not, the metadirective can be resolved early during compilation.) libgomp/ChangeLog: PR middle-end/118694 * testsuite/libgomp.fortran/metadirective-1.f90: xfail when compiling (also) for nvptx offloading as an error is then expected.
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694
--- Comment #12 from GCC Commits ---
The releases/gcc-15 branch has been updated by Tobias Burnus
:
https://gcc.gnu.org/g:d390c7e5bd03490485a0b036add096e2e8b811b9
commit r15-9731-gd390c7e5bd03490485a0b036add096e2e8b811b9
Author: Tobias Burnus
Date: Mon May 26 19:50:40 2025 +0200
c-c++-common/gomp/{attrs-,}metadirective-3.c: Fix expected result
[PR118694]
With compilation for nvptx enabled, two issues showed up:
(a) "error: 'target' construct with nested 'teams' construct contains
directives outside of the 'teams' construct"
See PR comment 9 why this is difficult to fix.
Solution: Add dg-bogus and accept/expect the error for 'target
offload_nvptx'.
(b) The assumptions about the dump for 'target offload_nvptx' were wrong
as the metadirective was already expanded to a OMP_NEXT_VARIANT
construct such that no 'omp metadirective' was left in either case.
Solution: Check that no 'omp metadirective' is left; additionally, expect
either OMP_NEXT_VARIANT (when offload_nvptx is available) or no 'teams'
directive at all (if not).
gcc/testsuite/ChangeLog:
PR middle-end/118694
* c-c++-common/gomp/attrs-metadirective-3.c: Change to never
expect 'omp metadirective' in the dump. If !offload_nvptx, check
that no 'teams' shows up in the dump; for offload_nvptx, expect
OMP_NEXT_VARIANT and an error about directive between 'target'
and 'teams'.
* c-c++-common/gomp/metadirective-3.c: Likewise.
(cherry picked from commit 5d6ed6d604ff949b650e48fa4eaed3ec8b6489c1)
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694
--- Comment #11 from GCC Commits ---
The master branch has been updated by Tobias Burnus :
https://gcc.gnu.org/g:5d6ed6d604ff949b650e48fa4eaed3ec8b6489c1
commit r16-883-g5d6ed6d604ff949b650e48fa4eaed3ec8b6489c1
Author: Tobias Burnus
Date: Mon May 26 19:50:40 2025 +0200
c-c++-common/gomp/{attrs-,}metadirective-3.c: Fix expected result
[PR118694]
With compilation for nvptx enabled, two issues showed up:
(a) "error: 'target' construct with nested 'teams' construct contains
directives outside of the 'teams' construct"
See PR comment 9 why this is difficult to fix.
Solution: Add dg-bogus and accept/expect the error for 'target
offload_nvptx'.
(b) The assumptions about the dump for 'target offload_nvptx' were wrong
as the metadirective was already expanded to a OMP_NEXT_VARIANT
construct such that no 'omp metadirective' was left in either case.
Solution: Check that no 'omp metadirective' is left; additionally, expect
either OMP_NEXT_VARIANT (when offload_nvptx is available) or no 'teams'
directive at all (if not).
gcc/testsuite/ChangeLog:
PR middle-end/118694
* c-c++-common/gomp/attrs-metadirective-3.c: Change to never
expect 'omp metadirective' in the dump. If !offload_nvptx, check
that no 'teams' shows up in the dump; for offload_nvptx, expect
OMP_NEXT_VARIANT and an error about directive between 'target'
and 'teams'.
* c-c++-common/gomp/metadirective-3.c: Likewise.
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694
--- Comment #10 from GCC Commits ---
The releases/gcc-15 branch has been updated by Tobias Burnus
:
https://gcc.gnu.org/g:cf619d4a366ad428421fdb7ad617b4749799cf93
commit r15-9730-gcf619d4a366ad428421fdb7ad617b4749799cf93
Author: Tobias Burnus
Date: Fri May 23 11:30:48 2025 +0200
libgomp.c-c++-common/metadirective-1.c: Expect 'error:' for nvptx compile
[PR118694]
OpenMP's 'target teams' is strictly coupled with 'teams'; if the latter
exists, the kernel is launched in directly with multiple teams. Thus,
the host has to know whether the teams construct exists or not. For
#pragma omp target
#pragma omp metadirective when (device={arch("nvptx")}: teams loop)
it is simple when 'nvptx' offloading is not supported, otherwise it depends
on the default device at runtime as the user code asks for a single team
for
host fallback and gcn offload and multiple for nvptx offload.
In any case, this commit ensures that no FAIL is printed, whatever a
future solution might look like. Instead of a dg-bogus combined with an
'xfail offload_target_nvptx', one an also argue that a dg-error for
'target offload_target_nvptx' would be more appropriate.
libgomp/ChangeLog:
PR middle-end/118694
* testsuite/libgomp.c-c++-common/metadirective-1.c: xfail when
compiling (also) for nvptx offloading as an error is then expected.
(cherry picked from commit b3d07ec7ac2ccd935a79b29e1a0e2eb16225286a)
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #9 from Tobias Burnus --- STATUS: * 'target teams' handles the target part differently, depending whether a 'teams' follows or not. Thus, the host (launching the offload kernel) has to know whether a 'teams' follows or not. * If the metadirective is placed after 'target' and inserts the 'teams' depending a 'device' context selector, this becomes a runtime property. If, e.g. only 'arch(nvptx)' uses 'teams', the runtime has to start the host fallback and amdgcn without teams but nvptx with teams; thus, there is a runtime dependency even if on the device side it can be resolved at compile time. On the user side, the best is to replace: omp target omp metadirective when(device=... : teams) // block by omp metadirective when(target_device=... : target teams // block such that at runtime the target_device kind is checked for. On the compiler side, GCC could print a better error message - possibly suggesting the change above - or if this becomes an important issue, GCC could also implement this handling change.
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694
--- Comment #8 from GCC Commits ---
The master branch has been updated by Tobias Burnus :
https://gcc.gnu.org/g:b3d07ec7ac2ccd935a79b29e1a0e2eb16225286a
commit r16-838-gb3d07ec7ac2ccd935a79b29e1a0e2eb16225286a
Author: Tobias Burnus
Date: Fri May 23 11:30:48 2025 +0200
libgomp.c-c++-common/metadirective-1.c: Expect 'error:' for nvptx compile
[PR118694]
OpenMP's 'target teams' is strictly coupled with 'teams'; if the latter
exists, the kernel is launched in directly with multiple teams. Thus,
the host has to know whether the teams construct exists or not. For
#pragma omp target
#pragma omp metadirective when (device={arch("nvptx")}: teams loop)
it is simple when 'nvptx' offloading is not supported, otherwise it depends
on the default device at runtime as the user code asks for a single team
for
host fallback and gcn offload and multiple for nvptx offload.
In any case, this commit ensures that no FAIL is printed, whatever a
future solution might look like. Instead of a dg-bogus combined with an
'xfail offload_target_nvptx', one an also argue that a dg-error for
'target offload_target_nvptx' would be more appropriate.
libgomp/ChangeLog:
PR middle-end/118694
* testsuite/libgomp.c-c++-common/metadirective-1.c: xfail when
compiling (also) for nvptx offloading as an error is then expected.
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #7 from sandra at gcc dot gnu.org --- Hmmm, that is helpful. Which pass is it that depends on the strict nesting of "omp teams" within "omp target" for code generation? Is that also in omp-low (where the nesting error is diagnosed) or some later pass?
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694
--- Comment #6 from Tobias Burnus ---
> Are we required to diagnose this as an error
> or is it allowable to permit this as an extension?
Answer "no" and "yes" - but the problem is that in general it does not work.
(Potential wrong code issues, albeit as below only bad and not very bad.)
The problem is that for
#pragma omp target
#pragma teams
on common offload hardware (like the AMD and Nvidia GPUs), the code is
not executed as:
- Start the offload kernel
- then, on the device start multiple teams
But:
- Start the offload kernel with several teams
Thus, it is impossible to add code between 'target' and 'teams'. For user code,
this is really very easy to generate wrong code in this case! And there is also
no directive that syncs multiple teams (contention groups) - except that once
the teams construct has finished, all teams have finished. Otherwise, only
thread synchronization in a single team (contention group works → cgroup ~
pteam).
* * *
In particular, if I execute your code and set the GOMP_DEBUG=1 env var, I see:
GOMP_OFFLOAD_run: kernel f$_omp_fn$0: launch [(teams: 1), 1, 1] [(lanes: 32),
(threads: 8), 1]
GOMP_OFFLOAD_run: kernel f$_omp_fn$0: launch [(teams: 1), 1, 1] [(lanes: 32),
(threads: 8), 1]
If I replace the condition by
when (user={condition(1)}: teams loop) \
the result is (twice):
GOMP_OFFLOAD_run: kernel f$_omp_fn$0: launch [(teams: 60), 1, 1] [(lanes:
32), (threads: 8), 1]
Thus, the code runs with 60 teams while your code is run only with a single
team. This implies that the the runtime could be 60 faster, cutting it down
from 1 hour to 1 minute!
(It could be even a bit faster if memory access + caching improve, but
typically it is quite a bit less than 60×, but it should still be very
visible!)
* * *
BTW: I wonder why it doesn't work for
constexpr int flag2 = 1;
...
when (user={condition(flag2)}: teams loop) \
shouldn't this be compile-time optimized (assuming C23 or a semi-new C++)?
I still see:
teams2.c:10:11: warning: ‘target’ construct with nested ‘teams’ construct
contains directives outside of the ‘teams’ construct [-Wopenmp]
and for the launch accordingly: "launch [(teams: 1),".
And, unsurprisingly, I also get the same for:
when (device={kind(nohost)}: teams loop) \
which is the case that we want to handle here.
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #5 from sandra at gcc dot gnu.org --- Created attachment 61485 --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=61485&action=edit test case metadirective-1a.c with dynamic selector Hmmm, interesting. I temporarily #ifdef'ed out the code that generates the bogus error. Not only does the libgomp metadirective-1.c test seem to run correctly, this similar test case with a dynamic selector (that implicitly inserts code to test the condition between the #pragma omp target and #pragma omp teams) works too. Are we required to diagnose this as an error or is it allowable to permit this as an extension? Diagnosing it would be rough; in general it can't happen until late resolution, and as I found previously, the optimizers that run before then can rewrite the code in all sorts of ways that make it hard to map it back onto its original structure. I'm thinking doing the check for correct target(?)/metadirective(?)/teams nesting either in the front ends or during gimplification, when we still have it in tree structure as close as possible to what the user wrote, would be better than omp-low, when gimplification has already messed with it.
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #4 from sandra at gcc dot gnu.org --- The nesting error is diagnosed in omp-low.cc. By this time the gimplifier has already replaced the metadirective with a conditional. Since it's a static selector, it'll be resolved to a constant during the ompdevlow pass, allowing the conditional to be optimized away. This can only happen when the metadirective selector is static and involves a "device" selector in a configuration where it *might* match. Dynamic selectors are not allowed, and the other case for static selectors that requires late resolution (a "construct" selector that can't be scored) doesn't apply here because of the enclosing #omp target directive. I thought briefly about refactoring the code in the gimplifier to duplicate the enclosing #omp target into the arms of the conditional, but that would screw up the resolution of the "device" selector and I don't think it's always statically resolvable by examining the clauses on the #omp target. I need to do some more investigation to discover if it will generate correct code if the bogus error is disabled. In that case the diagnostic could be moved to the gimplifier, or the gimplifier could emit some sort of new marker to enclose the metadirective expansion, without otherwise changing the expansion. Otherwise I have no ideas as yet. :-(
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #3 from Tobias Burnus --- On mainline currently failing (for a GCC configured with nvptx offloading): libgomp.c-c++-common/metadirective-1.c:10:11: error: 'target' construct with nested 'teams' construct contains directives outside of the 'teams' construct
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #2 from Tobias Burnus --- gcc/testsuite/gfortran.dg/gomp/metadirective-3.f90 also contains a testcase that triggers (if compiling with offload_nvptx enabled; xfailed).
[Bug middle-end/118694] OpenMP: target/metadirective/teams directive nesting gives error
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118694 --- Comment #1 from sandra at gcc dot gnu.org --- I see the spec does, in fact, prohibit metadirectives that expand into dynamic selector code here -- it's at the bottom of page 324 of the OpenMP 6.0 document. So the problem is just the late resolution expansion produced by the gimplifier when some selectors cannot be matched and/or scored yet.
