[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/rafaelauler approved this pull request. lgtm https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov edited https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov edited https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov ready_for_review https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov edited https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
llvmbot wrote:
@llvm/pr-subscribers-bolt
Author: Amir Ayupov (aaupov)
Changes
Call continuation logic relies on assumptions about fall-through origin:
- the branch is external to the function,
- fall-through start is at the beginning of the block,
- the block is not an entry point or a landing pad.
Leverage trace information to explicitly check whether the origin is a
return instruction, and defer to checks above only in case of
DSO-external branch source.
This covers both regular and BAT cases, addressing call continuation
fall-through undercounting in the latter mode.
Depends on #143289.
Test Plan: updated callcont-fallthru.s
---
Full diff: https://github.com/llvm/llvm-project/pull/143295.diff
5 Files Affected:
- (modified) bolt/include/bolt/Profile/BoltAddressTranslation.h (+2-1)
- (modified) bolt/include/bolt/Profile/DataAggregator.h (+9-3)
- (modified) bolt/lib/Profile/BoltAddressTranslation.cpp (+8-2)
- (modified) bolt/lib/Profile/DataAggregator.cpp (+41-30)
- (modified) bolt/test/X86/callcont-fallthru.s (+40-32)
``diff
diff --git a/bolt/include/bolt/Profile/BoltAddressTranslation.h
b/bolt/include/bolt/Profile/BoltAddressTranslation.h
index fcc578f35e322..917531964e9b6 100644
--- a/bolt/include/bolt/Profile/BoltAddressTranslation.h
+++ b/bolt/include/bolt/Profile/BoltAddressTranslation.h
@@ -103,7 +103,8 @@ class BoltAddressTranslation {
/// otherwise.
std::optional getFallthroughsInTrace(uint64_t FuncAddress,
uint64_t From,
- uint64_t To) const;
+ uint64_t To,
+ bool IsReturn) const;
/// If available, fetch the address of the hot part linked to the cold part
/// at \p Address. Return 0 otherwise.
diff --git a/bolt/include/bolt/Profile/DataAggregator.h
b/bolt/include/bolt/Profile/DataAggregator.h
index 10d96fbeca3e2..96969cf53baca 100644
--- a/bolt/include/bolt/Profile/DataAggregator.h
+++ b/bolt/include/bolt/Profile/DataAggregator.h
@@ -132,6 +132,9 @@ class DataAggregator : public DataReader {
/// and use them later for processing and assigning profile.
std::unordered_map TraceMap;
std::vector> Traces;
+ /// Pre-populated addresses of returns, coming from pre-aggregated data or
+ /// disassembly. Used to disambiguate call-continuation fall-throughs.
+ std::unordered_set Returns;
std::unordered_map BasicSamples;
std::vector MemSamples;
@@ -204,8 +207,8 @@ class DataAggregator : public DataReader {
/// Return a vector of offsets corresponding to a trace in a function
/// if the trace is valid, std::nullopt otherwise.
std::optional, 16>>
- getFallthroughsInTrace(BinaryFunction &BF, const Trace &Trace,
- uint64_t Count) const;
+ getFallthroughsInTrace(BinaryFunction &BF, const Trace &Trace, uint64_t
Count,
+ bool IsReturn) const;
/// Record external entry into the function \p BF.
///
@@ -265,11 +268,14 @@ class DataAggregator : public DataReader {
uint64_t From, uint64_t To, uint64_t Count,
uint64_t Mispreds);
+ /// Checks if \p Addr corresponds to a return instruction.
+ bool checkReturn(uint64_t Addr);
+
/// Register a \p Branch.
bool doBranch(uint64_t From, uint64_t To, uint64_t Count, uint64_t Mispreds);
/// Register a trace between two LBR entries supplied in execution order.
- bool doTrace(const Trace &Trace, uint64_t Count);
+ bool doTrace(const Trace &Trace, uint64_t Count, bool IsReturn);
/// Parser helpers
/// Return false if we exhausted our parser buffer and finished parsing
diff --git a/bolt/lib/Profile/BoltAddressTranslation.cpp
b/bolt/lib/Profile/BoltAddressTranslation.cpp
index a253522e4fb15..732229b52c221 100644
--- a/bolt/lib/Profile/BoltAddressTranslation.cpp
+++ b/bolt/lib/Profile/BoltAddressTranslation.cpp
@@ -511,8 +511,8 @@ uint64_t BoltAddressTranslation::translate(uint64_t
FuncAddress,
std::optional
BoltAddressTranslation::getFallthroughsInTrace(uint64_t FuncAddress,
- uint64_t From,
- uint64_t To) const {
+ uint64_t From, uint64_t To,
+ bool IsReturn) const {
SmallVector, 16> Res;
// Filter out trivial case
@@ -530,6 +530,12 @@ BoltAddressTranslation::getFallthroughsInTrace(uint64_t
FuncAddress,
auto FromIter = Map.upper_bound(From);
if (FromIter == Map.begin())
return Res;
+
+ // For fall-throughs originating at returns, go back one entry to cover call
+ // site.
+ if (IsReturn)
+--FromIter;
+
// Skip instruction entries, to create fallthroughs we are only interested in
// BB boundaries
do {
diff --git a/bolt/lib/Profile/DataA
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov edited https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov updated https://github.com/llvm/llvm-project/pull/143295 ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [BOLT] Explicitly check for returns when extending call continuation profile (PR #143295)
https://github.com/aaupov created https://github.com/llvm/llvm-project/pull/143295 Call continuation logic relies on assumptions about fall-through origin: - the branch is external to the function, - fall-through start is at the beginning of the block, - the block is not an entry point or a landing pad. Leverage trace information to explicitly check whether the origin is a return instruction, and defer to checks above only in case of DSO-external branch source. This covers both regular and BAT cases, addressing call continuation fall-through undercounting in the latter mode. Test Plan: TBD ___ llvm-branch-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
