Hi Sandra,
Hi Sandra, hi all,
Sandra Loosemore wrote:
Regarding the documentation (current version at ...)
https://gcc.gnu.org/onlinedocs/gcc/OpenMP-and-OpenACC-Options.html#index-fopenmp-ompt
Can you first have a look at the current version, just as comparison.
My proposal is:
-fopenmp-ompt
For some OpenMP constructs, generate calls before and after the
region instead of only on region entry. This improves the OMPT
tracing at expense of additional calls. In particular, the
additional calls will be added for static worksharing loops,
static worksharing @code{dispatch}, and the @code{single},
@code{masked}, and @code{master} construct.
Requires @option{-fopenmp}.
-fopenmp-ompt-details
Add additional calls into the OpenMP runtime that might affect
the performance but further improves OMPT tracing. In particular,
dispatch callbacks are generated at the beginning of each
workshare chunk for static worksharing loops and
static worksharing @code{dispatch}. This option implies
@option{-fopenmp-ompt}. Requires @option{-fopenmp}.
There are three levels of information: First, those are about OpenMP and
OMPT (well, the name already states so) – and adding the flag improves
the data obtained by OMPT but comes at a cost.
Secondly, -fopenmp-ompt will change (some) "region starts (no idea when
it ends)" to "region start" + "region ends" events.
Thirdly, this happens for the listed constructs.
The last item is really for the experts that care about the fine print.
[Similarly for -fopenmp-ompt-details: More details, but a lot more
overhead!]
* * *
I have very little state on OMPT at present,
[See also post script at the bottom]
The OMPT – 'T' means tool – interface permits a tool to register
callbacks. Thus, when the program encounters, e.g. an 'omp parallel' –
and the tool has registered a callback, the libgomp runtime will call
the callback with the information that the event that a parallel region
has been entered.
The tracing tool already sees which threads exist, which wait and which
run — it also usually sees the function name. However, it can be helpful
to know that e.g. an 'omp parallel' caused that more threads are
spawned – or the size of a loop when looking where the time is spent.
OMPT makes this possible.
(Technically, for OMPT, the tool ships with a small shared library which
the OpenMP runtime loads if specified in the OMP_TOOLS environment
variable; the OpenMP runtime calls then a function in that library and
in turn that library registers the events of interests it wants to get
notified about.)
* * *
There is some trade of between providing more details about the program
– making tracing more useful – and the performance penalty. — In
particular, calls into the library should be avoided if not needed.
Thus, the idea of -fopenmp-ompt and -fopenmp-ompt-details adds more
runtime calls. With both flags, calls are added that are no ops, unless
a tool has registered an event – then the additional data helps.
Example: By default, GCC only sends an event for 'omp masked' (and the
tool gets the event 'startstop') — but with -fopenmp-ompt it gets two
events: one for 'omp masked' (event 'start') and one for 'omp end masked
(event 'end').
* * *
and I didn't understand the terminology in Tobias's proposed text
(e.g. "static worksharing loops", "workshare chunk").
First, I was mostly taking the current wording – see link above; that's
in the OpenMP section and you find more in the OpenMP specification (or
a book about OpenMP). In particular, if you pick, e.g., OpenMP 6.0,
https://www.openmp.org/specifications/ you will find:
"13 Work-Distribution Constructs" which consist of:
single, scope, sections, workshare, workdistribute, worksharing-loop
(C/C++'s "for" + Fortran's "do"), distribute, and loop.
When now looking at "13.6 Worksharing-Loop Constructs" – that's 'omp
for' and 'omp do' — a 'schedule' clause exists with the permitted values
'static', 'dynamic', and 'guided' (+ 'auto' and 'runtime').
And per "TABLE 13.1: 'work' OMPT types for Worksharing-Loop": OpenMP's
OMPT actually tells the tool which scheduling method was taken
[events: ompt_work_loop_{static,dynamic,guided} plus unknown
(ompt_work_loop) and: some other method alias implementation defined
("ompt_work_loop_other").
GCC already supports a bunch of scheduling methods – some already
require multiple calls into the run time, others like static only one at
the beginning. — Thus, for 'static' there -fopenmp-ompt changes the
single call to two calls, one at the beginning and one at the end of the
region.
Similarly for the 'distribute' construct — one can specify
'dyn_schedule(static)' or not specify it. – And for 'static' again only
one runtime call is needed – and -fopenmp-ompt turns this into two.
* * *
For 'chunk', I refer to the glossary; OpenMP 6 defines it as:
"chunk — A contiguous non-empty subset of the collapsed iterations of a
loop-collapsing construct"
The tool gets more information when it knows when a new chunk is
started, but that has a way higher overhead – hence, that call is only
added with -fopenmp-ompt-details.
Namely: "A thread dispatches a registered dispatch callback for each
occurrence of a ws-loop-iteration-begin or ws-loop-chunk-begin event in
that thread."
with
"The ws-loop-iteration-begin event occurs at the beginning of each
collapsed iteration of a worksharing-loop region. The
ws-loop-chunk-begin event occurs for each scheduled chunk of
a worksharing-loop region before the implicit task executes any of the
collapsed iterations."
For a simple loop on huge data that just does some assignments, the
overhead is much higher than the actual work. — While when doing
something complex in the loop, it might be just noise. — In any case, it
can be helpful in some cases, but usually the -fopenmp-ompt-details is
to be avoided for performance reasons. (Plus: If the tool actually
stores all the events, the tracing data quickly becomes really large.)
* * *
@PA/@Thomas – BTW: I wonder whether we also need a begin/end for
'scope'? — I think we need it. (GCC only calls it if used with
reductions. OpenMP defines the callbacks ompt_scope_begin and
ompt_scope_end with type ompt_work_scope.)
* * *
I don't know if that is a bug in the text or if I just need to read
something that defines those terms -- but in that case GCC users need
to be pointed at the appropriate reference too, not just me.
I think a user who runs a program with some tracing tool has some idea
about this – and the generic: will add an event at the beginning/end of
the region instead of only at the beginning (→ more details a bit
slower) is essentially what one needs to know. If one is an expect,
knowing what exactly is covered makes sense, but if they are already
OpenMP/OMPT experts, they should know the words.
In any case, IMHO the new wording is clearer or non experts than the
current one, not that it couldn't be improved more.
Tobias
PS: The typical way one looks at tracing data is as timeline where each
process (MPI) or thread (OpenMP) has a row – and one can zoom in or
drill in. For instance:*
https://vampir.eu/tutorial/manual/performance_data_visualization#sec-master_process_timeline
If something looks interesting, one has a closer look – and then it
helps to know what is a parallel region, how many threads were requested
and actually used – what's the iteration count of a loop etc. — The bare
data the tracing tool obtains itself – but for such details, it needs
help – provided by events send by the OpenMP runtime.
(*There are plenty of tools, some only visualize – others only measure,
some do both. Some are free ohers not. I just picked one with a somewhat
helpful screenshot.)
