https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126241
Bug ID: 126241
Summary: Missed devirtualization of immutable std::function
dispatch blocks
Product: gcc
Version: 17.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: ipa
Assignee: unassigned at gcc dot gnu.org
Reporter: ptomsich at gcc dot gnu.org
Target Milestone: ---
Created attachment 65027
--> https://gcc.gnu.org/bugzilla/attachment.cgi?id=65027&action=edit
self-contained reproducer (459.cxx)
When a std::function object's target is fixed for the lifetime of the object
(constructed once from a known callable, the functor field never rewritten), a
call through it still requires a two-level indirect dispatch:
call f(args)
-> blr _M_invoker ; std::function::operator()
-> _Function_handler<...>::_M_invoke
-> blr <functor> ; the real callable
Both indirect targets are statically determined, but GCC does not fold the
dispatch to a direct call of the underlying callable. The functor pointer is
stored by the std::function constructor and loaded by _M_invoke (a different
function), so this is an inter-procedural optimisation. The store goes through
the type-punned _Any_data, which value-numbering does not forward to the load.
The two-level dispatch therefore survives as an indirect call, and the
vectorizer refuses to vectorise any loop containing it.
Nothing in this is specific to std::function. Any type-erased wrapper of the
same shape (a single forwarder dispatching through an immutable
function-pointer field) hits the same limitation.
*Motivating case (772.marian_r, src/marian/functional/operators.h)*
static inline float32x4
loop4(const std::function<float(const float&, const float&)>& f,
const float32x4& x, const float32x4& y) {
float32x4 out;
for (int i = 0; i < 4; i++)
((float*)&out)[i] = f(((const float*)&x)[i], ((const float*)&y)[i]);
return out;
}
static float32x4 mul(const float32x4& x, const float32x4& y) {
return loop4(Ops<float>::mul, x, y); // known free fn -> std::function
}
Every element-wise op (add/mul/sub, ReLU, sigmoid, tanh, Softmax, ...) routes
through loop4(std::function, ...).
Self-contained reproducer (single TU, no LTO) attached.
Compile with (pick your -mcpu= appropriately):
g++ -c -O3 -mcpu=ampere1 \
--param early-inlining-insns=256 \
--param max-inline-insns-auto=128 \
--param inline-unit-growth=256 \
459.cxx
The three inline params only serve to shrink the reproducer to a single TU;
772.marian_r exposes the same un-lowered dispatch at stock -Ofast -flto with
default parameters.
Comparision before/after adding an experimental optimisation:
-fno-devirtualize-function-objects (current behaviour): 6 blr, 0 NEON .4s,
loop NOT vectorized
-fdevirtualize-function-objects (experimental): 0 blr, 2 NEON .4s,
loop vectorized
*Impact of recovering the devirt*
Measured on 772.marian_r (-Ofast -mcpu=ampere1a -flto=32 -funroll-loops, single
copy):
input with without delta
EuroPat 110.80 s 118.24 s +6.3%
TildeMODEL 72.49 s 79.22 s +8.5%
*Proposed fix (rationale for a new pass is below)*
A pre-IPA lowering pass (a SIMPLE_IPA pass scheduled after
pass_all_early_optimizations, before IPA-inline) that:
1. Recognizes the std::function _M_invoke forwarder name-free, via a small
extension to the ipa-modref summary (a flag recording that the callee performs
exactly one indirect call through a pointer loaded from parameter 0 and has no
other indirect call). modref does not track this today, we need an extra
parameter index per summary. No dependence on libstdc++ symbol names;
target-independent.
2. Recovers the functor pointer that value-numbering misses across the
constructor's (type-punned) _Any_data store, via get_ref_base_and_extent,
requiring a unique &FUNCTION_DECL store to the field.
3. Proves the dispatch object is not written across the call (a shared
immutability check, with a conservative bail-out if the address escapes) with a
strict signature-compatibility gate (<- yes, we shot ourselves in the foot
here, before...), so IPA-inline pulls in the body and the loop vectorizes
Under LTO the transform runs at compile time (LGEN), per TU, after early
inlining; the _M_invoke instantiation is a comdat in the same TU, so nothing
new needs streaming to WPA. The above numbers are from -ftlo builds.
*Pre-IPA vs IPA-CP*
We tried putting this in IPA-CP first, as there is a TODO for exactly this type
of optimisation:
ipa-cp.cc (propagate_bits/aggregate handling):
/* TODO: We currently do not handle member method pointers in IPA-CP
(we only use it for indirect inlining), we should propagate them too. */
However, we pushed up against a phase-ordering issue on marian_r:
- Even where IPA-CP did form a direct edge, the resulting direct remained a
call in the inner loop (i.e., a vectorizer region boundary) and the loop still
did not vectorize. To get the optimisation, we need the callable inlined, not
just a direct call edge.
- adding a new pass sidesteps this by running before IPA-inline: we can rewrite
the dispatch to a direct call *before* the inliner runs in the pipeline. The
callable body thus is inlined, the region boundary is gone, and the vectorizer
handles the rest.
Indirect inlining itself (the existing use the TODO refers to) does not catch
our case: the functor value is only known in the frame that constructed the
std::function, and the known-aggregate-contents computation at that call site
does not see through the type-punned _Any_data store.
The recognition (modref "single deferred dispatch through parameter 0"
analysis) can be kept a reusable function and shared with IPA-CP.
*Why not speculative devirtualization*
The existing -fdevirtualize-speculatively leaves a guard and an indirect
fallback inside the innermost loop => still a vectorizer region boundary.
*Why not early FRE + repeated early inlining*
That requires iterating early inlining and the early optimizations to a
fixpoint. It also pushes up against the type-punned _Any_data load, which
SCCVN does not forward even within a single function. The proposed pass does
the rewrite in one linear walk without ever inlining the forwarder and dropping
the _Any_data argument.
*Related*
PR91771 is the closest existing report: it is a vtable dispatch (OBJ_TYPE_REF)
and the target is derivable from the type by ipa-devirt. Our case is a plain
indirect call through a data field and only derivable from dataflow (not for
ipa-devirt in our reading).
