modimo created this revision.
modimo added reviewers: wenlei, asbirlea, vsk.
modimo added a project: LLVM.
Herald added subscribers: llvm-commits, cfe-commits, dexonsmith, steven_wu,
hiraditya.
Herald added a project: clang.
modimo requested review of this revision.
D65060 <https://reviews.llvm.org/D65060> uncovered that trying to use BFI in
loop passes can lead to non-deterministic behavior when blocks are re-used
while retaining old BFI data.
To make sure BFI is preserved through loop passes a Value Handle (VH) callback
is registered on blocks themselves. When a block is freed it now also wipes out
the accompanying BFI entry such that stale BFI data can no longer persist
resolving the determinism issue.
An optimistic approach would be to incrementally update BFI information
throughout the loop passes rather than only invalidating them on removed
blocks. The issues with that are:
1. It is not clear how BFI information should be incrementally updated: If a
block is duplicated does its BFI information come with? How about if it's
split/modified/moved around?
2. Assuming we can address these problems the implementation here will be a
massive undertaking.
There's a known need of BFI in LICM analysis which requires correct but not
incrementally updated BFI data. Other loop passes can now also use this level
of information and richer updates can be made as needed.
This diff also moves BFI to be a part of LoopStandardAnalysisResults since the
previous method using getCachedResults now (correctly!) statically asserts
(D72893 <https://reviews.llvm.org/D72893>) that this data isn't static through
the loop passes.
Testing
Ninja check
Repository:
rG LLVM Github Monorepo
https://reviews.llvm.org/D86156
Files:
clang/test/CodeGen/thinlto-distributed-newpm.ll
llvm/include/llvm/Analysis/BlockFrequencyInfo.h
llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h
llvm/include/llvm/Analysis/LoopAnalysisManager.h
llvm/include/llvm/Transforms/Scalar/LoopPassManager.h
llvm/lib/Analysis/BlockFrequencyInfo.cpp
llvm/lib/Transforms/Scalar/LoopDistribute.cpp
llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp
llvm/lib/Transforms/Utils/LoopVersioning.cpp
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
llvm/test/Other/loop-pm-invalidation.ll
llvm/test/Other/new-pass-manager.ll
llvm/test/Other/new-pm-defaults.ll
llvm/test/Other/new-pm-thinlto-defaults.ll
llvm/test/Other/pass-pipelines.ll
llvm/test/Transforms/LoopRotate/pr35210.ll
llvm/unittests/Transforms/Scalar/LoopPassManagerTest.cpp
Index: llvm/unittests/Transforms/Scalar/LoopPassManagerTest.cpp
===================================================================
--- llvm/unittests/Transforms/Scalar/LoopPassManagerTest.cpp
+++ llvm/unittests/Transforms/Scalar/LoopPassManagerTest.cpp
@@ -10,6 +10,9 @@
#include "llvm/Analysis/AliasAnalysis.h"
#include "llvm/Analysis/AssumptionCache.h"
#include "llvm/Analysis/MemorySSA.h"
+#include "llvm/Analysis/BlockFrequencyInfo.h"
+#include "llvm/Analysis/BranchProbabilityInfo.h"
+#include "llvm/Analysis/PostDominators.h"
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/Analysis/TargetTransformInfo.h"
@@ -294,6 +297,9 @@
// those.
FAM.registerPass([&] { return AAManager(); });
FAM.registerPass([&] { return AssumptionAnalysis(); });
+ FAM.registerPass([&] { return BlockFrequencyAnalysis(); });
+ FAM.registerPass([&] { return BranchProbabilityAnalysis(); });
+ FAM.registerPass([&] { return PostDominatorTreeAnalysis(); });
FAM.registerPass([&] { return MemorySSAAnalysis(); });
FAM.registerPass([&] { return ScalarEvolutionAnalysis(); });
FAM.registerPass([&] { return TargetLibraryAnalysis(); });
Index: llvm/test/Transforms/LoopRotate/pr35210.ll
===================================================================
--- llvm/test/Transforms/LoopRotate/pr35210.ll
+++ llvm/test/Transforms/LoopRotate/pr35210.ll
@@ -19,12 +19,15 @@
; CHECK-NEXT: Running analysis: TargetLibraryAnalysis on f
; CHECK-NEXT: Running analysis: ScalarEvolutionAnalysis on f
; CHECK-NEXT: Running analysis: TargetIRAnalysis on f
+; CHECK-NEXT: Running analysis: BlockFrequencyAnalysis on f
+; CHECK-NEXT: Running analysis: BranchProbabilityAnalysis on f
; CHECK-NEXT: Running analysis: InnerAnalysisManagerProxy{{.*}} on f
; CHECK-NEXT: Starting Loop pass manager run.
; CHECK-NEXT: Running pass: LoopRotatePass on Loop at depth 1 containing: %bb<header><exiting>,%bb4<latch>
; CHECK-NEXT: Folding loop latch bb4 into bb
; CHECK-NEXT: Finished Loop pass manager run.
; CHECK-NEXT: Invalidating analysis: PostDominatorTreeAnalysis on f
+; CHECK-NEXT: Invalidating analysis: BranchProbabilityAnalysis on f
; CHECK-NEXT: Running pass: ADCEPass on f
; CHECK-NEXT: Running analysis: PostDominatorTreeAnalysis on f
; CHECK-NEXT: Finished llvm::Function pass manager run.
@@ -44,12 +47,15 @@
; MSSA-NEXT: Running analysis: TargetLibraryAnalysis on f
; MSSA-NEXT: Running analysis: ScalarEvolutionAnalysis on f
; MSSA-NEXT: Running analysis: TargetIRAnalysis on f
+; MSSA-NEXT: Running analysis: BlockFrequencyAnalysis on f
+; MSSA-NEXT: Running analysis: BranchProbabilityAnalysis on f
; MSSA-NEXT: Running analysis: InnerAnalysisManagerProxy{{.*}} on f
; MSSA-NEXT: Starting Loop pass manager run.
; MSSA-NEXT: Running pass: LoopRotatePass on Loop at depth 1 containing: %bb<header><exiting>,%bb4<latch>
; MSSA-NEXT: Folding loop latch bb4 into bb
; MSSA-NEXT: Finished Loop pass manager run.
; MSSA-NEXT: Invalidating analysis: PostDominatorTreeAnalysis on f
+; MSSA-NEXT: Invalidating analysis: BranchProbabilityAnalysis on f
; MSSA-NEXT: Running pass: ADCEPass on f
; MSSA-NEXT: Running analysis: PostDominatorTreeAnalysis on f
; MSSA-NEXT: Finished llvm::Function pass manager run.
Index: llvm/test/Other/pass-pipelines.ll
===================================================================
--- llvm/test/Other/pass-pipelines.ll
+++ llvm/test/Other/pass-pipelines.ll
@@ -54,7 +54,7 @@
; CHECK-O2-NOT: Manager
; CHECK-O2: Loop Pass Manager
; CHECK-O2: Loop Pass Manager
-; CHECK-O2-NOT: Manager
+; Requiring block frequency for LICM will place ICM and rotation under separate Loop Pass Manager
; FIXME: We shouldn't be pulling out to simplify-cfg and instcombine and
; causing new loop pass managers.
; CHECK-O2: Simplify the CFG
Index: llvm/test/Other/new-pm-thinlto-defaults.ll
===================================================================
--- llvm/test/Other/new-pm-thinlto-defaults.ll
+++ llvm/test/Other/new-pm-thinlto-defaults.ll
@@ -129,6 +129,9 @@
; CHECK-O-NEXT: Running pass: LCSSAPass
; CHECK-O-NEXT: Finished llvm::Function pass manager run
; CHECK-O-NEXT: Running analysis: ScalarEvolutionAnalysis
+; CHECK-O-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-O-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
; CHECK-O-NEXT: Starting Loop pass manager run.
; CHECK-O-NEXT: Running pass: LoopInstSimplifyPass
@@ -184,7 +187,6 @@
; CHECK-O23SZ-NEXT: Finished llvm::Function pass manager run
; CHECK-O23SZ-NEXT: Running pass: LICMPass on Loop at depth 1 containing: %loop
; CHECK-O-NEXT: Running pass: ADCEPass
-; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-O-NEXT: Running pass: SimplifyCFGPass
; CHECK-O-NEXT: Running pass: InstCombinePass
; CHECK-O-NEXT: Finished llvm::Function pass manager run.
@@ -210,8 +212,6 @@
; CHECK-POSTLINK-O-NEXT: Running pass: LoopDistributePass
; CHECK-POSTLINK-O-NEXT: Running pass: InjectTLIMappings
; CHECK-POSTLINK-O-NEXT: Running pass: LoopVectorizePass
-; CHECK-POSTLINK-O-NEXT: Running analysis: BlockFrequencyAnalysis
-; CHECK-POSTLINK-O-NEXT: Running analysis: BranchProbabilityAnalysis
; CHECK-POSTLINK-O-NEXT: Running pass: LoopLoadEliminationPass
; CHECK-POSTLINK-O-NEXT: Running analysis: LoopAccessAnalysis
; CHECK-POSTLINK-O-NEXT: Running pass: InstCombinePass
Index: llvm/test/Other/new-pm-defaults.ll
===================================================================
--- llvm/test/Other/new-pm-defaults.ll
+++ llvm/test/Other/new-pm-defaults.ll
@@ -165,6 +165,9 @@
; CHECK-O-NEXT: Running pass: LCSSAPass
; CHECK-O-NEXT: Finished llvm::Function pass manager run.
; CHECK-O-NEXT: Running analysis: ScalarEvolutionAnalysis
+; CHECK-O-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-O-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
; CHECK-O-NEXT: Starting Loop pass manager run.
; CHECK-O-NEXT: Running pass: LoopInstSimplifyPass
@@ -212,7 +215,6 @@
; CHECK-O23SZ-NEXT: Running pass: LICMPass
; CHECK-EP-SCALAR-LATE-NEXT: Running pass: NoOpFunctionPass
; CHECK-O-NEXT: Running pass: ADCEPass
-; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-O-NEXT: Running pass: SimplifyCFGPass
; CHECK-O-NEXT: Running pass: InstCombinePass
; CHECK-EP-PEEPHOLE-NEXT: Running pass: NoOpFunctionPass
@@ -240,8 +242,6 @@
; CHECK-O-NEXT: Running pass: LoopDistributePass
; CHECK-O-NEXT: Running pass: InjectTLIMappings
; CHECK-O-NEXT: Running pass: LoopVectorizePass
-; CHECK-O-NEXT: Running analysis: BlockFrequencyAnalysis
-; CHECK-O-NEXT: Running analysis: BranchProbabilityAnalysis
; CHECK-O-NEXT: Running pass: LoopLoadEliminationPass
; CHECK-O-NEXT: Running analysis: LoopAccessAnalysis
; CHECK-O-NEXT: Running pass: InstCombinePass
Index: llvm/test/Other/new-pass-manager.ll
===================================================================
--- llvm/test/Other/new-pass-manager.ll
+++ llvm/test/Other/new-pass-manager.ll
@@ -455,6 +455,9 @@
; CHECK-REPEAT-LOOP-PASS-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-REPEAT-LOOP-PASS-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-REPEAT-LOOP-PASS-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-REPEAT-LOOP-PASS-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-REPEAT-LOOP-PASS-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-REPEAT-LOOP-PASS-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-REPEAT-LOOP-PASS-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}>
; CHECK-REPEAT-LOOP-PASS-NEXT: Starting Loop pass manager run
; CHECK-REPEAT-LOOP-PASS-NEXT: Running pass: RepeatedPass
Index: llvm/test/Other/loop-pm-invalidation.ll
===================================================================
--- llvm/test/Other/loop-pm-invalidation.ll
+++ llvm/test/Other/loop-pm-invalidation.ll
@@ -67,6 +67,9 @@
; CHECK-LOOP-INV-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Loop
; CHECK-LOOP-INV-NEXT: Starting {{.*}}Loop pass manager run.
; CHECK-LOOP-INV-NEXT: Running pass: NoOpLoopPass
@@ -100,6 +103,9 @@
; CHECK-SCEV-INV-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Loop
; CHECK-SCEV-INV-NEXT: Starting {{.*}}Loop pass manager run.
; CHECK-SCEV-INV-NEXT: Running pass: NoOpLoopPass
@@ -143,6 +149,9 @@
; CHECK-LOOP-INV-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Loop
; CHECK-LOOP-INV-NEXT: Starting {{.*}}Loop pass manager run.
; CHECK-LOOP-INV-NEXT: Running pass: NoOpLoopPass
@@ -183,6 +192,9 @@
; CHECK-SCEV-INV-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Loop
; CHECK-SCEV-INV-NEXT: Starting {{.*}}Loop pass manager run.
; CHECK-SCEV-INV-NEXT: Running pass: NoOpLoopPass
@@ -240,6 +252,9 @@
; CHECK-LOOP-INV-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-LOOP-INV-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-LOOP-INV-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Loop
; CHECK-LOOP-INV-NEXT: Starting {{.*}}Loop pass manager run.
; CHECK-LOOP-INV-NEXT: Running pass: NoOpLoopPass
@@ -273,6 +288,9 @@
; CHECK-SCEV-INV-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-SCEV-INV-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-SCEV-INV-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Loop
; CHECK-SCEV-INV-NEXT: Starting {{.*}}Loop pass manager run.
; CHECK-SCEV-INV-NEXT: Running pass: NoOpLoopPass
@@ -302,12 +320,17 @@
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running analysis: TargetLibraryAnalysis
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running analysis: ScalarEvolutionAnalysis
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running analysis: TargetIRAnalysis
+; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running analysis: BlockFrequencyAnalysis
+; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running analysis: BranchProbabilityAnalysis
+; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running analysis: PostDominatorTreeAnalysis
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Loop
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Starting {{.*}}Loop pass manager run.
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running pass: NoOpLoopPass
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running pass: LoopDeletionPass
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Clearing all analysis results for:
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Finished {{.*}}Loop pass manager run.
+; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Invalidating analysis: PostDominatorTreeAnalysis
+; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Invalidating analysis: BranchProbabilityAnalysis
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Running pass: InvalidateAnalysisPass<{{.*}}ScalarEvolutionAnalysis
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Invalidating analysis: ScalarEvolutionAnalysis
; CHECK-SCEV-INV-AFTER-DELETE-NEXT: Invalidating analysis: InnerAnalysisManagerProxy<{{.*}}Loop
Index: llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
===================================================================
--- llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -196,7 +196,7 @@
cl::desc("Enable vectorization on interleaved memory accesses in a loop"));
/// An interleave-group may need masking if it resides in a block that needs
-/// predication, or in order to mask away gaps.
+/// predication, or in order to mask away gaps.
static cl::opt<bool> EnableMaskedInterleavedMemAccesses(
"enable-masked-interleaved-mem-accesses", cl::init(false), cl::Hidden,
cl::desc("Enable vectorization on masked interleaved memory accesses in a loop"));
@@ -4059,7 +4059,7 @@
auto *IncomingValue = LCSSAPhi.getIncomingValue(0);
// Non-instruction incoming values will have only one value.
unsigned LastLane = 0;
- if (isa<Instruction>(IncomingValue))
+ if (isa<Instruction>(IncomingValue))
LastLane = Cost->isUniformAfterVectorization(
cast<Instruction>(IncomingValue), VF)
? 0
@@ -5683,7 +5683,7 @@
}
}
}
-
+
for (auto& pair : RegUsage) {
if (MaxUsages[j].find(pair.first) != MaxUsages[j].end())
MaxUsages[j][pair.first] = std::max(MaxUsages[j][pair.first], pair.second);
@@ -5701,7 +5701,7 @@
for (unsigned i = 0, e = VFs.size(); i < e; ++i) {
SmallMapVector<unsigned, unsigned, 4> Invariant;
-
+
for (auto Inst : LoopInvariants) {
unsigned Usage = VFs[i] == 1 ? 1 : GetRegUsage(Inst->getType(), VFs[i]);
unsigned ClassID = TTI.getRegisterClassForType(VFs[i] > 1, Inst->getType());
@@ -5993,7 +5993,7 @@
// vectorized loop where the user of it is a vectorized instruction.
const Align Alignment = getLoadStoreAlignment(I);
Cost += VF * TTI.getMemoryOpCost(I->getOpcode(), ValTy->getScalarType(),
- Alignment, AS,
+ Alignment, AS,
TTI::TCK_RecipThroughput);
// Get the overhead of the extractelement and insertelement instructions
@@ -8386,7 +8386,8 @@
auto &LAM = AM.getResult<LoopAnalysisManagerFunctionProxy>(F).getManager();
std::function<const LoopAccessInfo &(Loop &)> GetLAA =
[&](Loop &L) -> const LoopAccessInfo & {
- LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE, TLI, TTI, MSSA};
+ LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE,
+ TLI, TTI, nullptr, MSSA};
return LAM.getResult<LoopAccessAnalysis>(L, AR);
};
auto &MAMProxy = AM.getResult<ModuleAnalysisManagerFunctionProxy>(F);
Index: llvm/lib/Transforms/Utils/LoopVersioning.cpp
===================================================================
--- llvm/lib/Transforms/Utils/LoopVersioning.cpp
+++ llvm/lib/Transforms/Utils/LoopVersioning.cpp
@@ -357,7 +357,8 @@
auto &LAM = AM.getResult<LoopAnalysisManagerFunctionProxy>(F).getManager();
auto GetLAA = [&](Loop &L) -> const LoopAccessInfo & {
- LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE, TLI, TTI, MSSA};
+ LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE,
+ TLI, TTI, nullptr, MSSA};
return LAM.getResult<LoopAccessAnalysis>(L, AR);
};
Index: llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp
===================================================================
--- llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp
+++ llvm/lib/Transforms/Scalar/LoopLoadElimination.cpp
@@ -705,7 +705,8 @@
auto &LAM = AM.getResult<LoopAnalysisManagerFunctionProxy>(F).getManager();
bool Changed = eliminateLoadsAcrossLoops(
F, LI, DT, BFI, PSI, [&](Loop &L) -> const LoopAccessInfo & {
- LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE, TLI, TTI, MSSA};
+ LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE,
+ TLI, TTI, nullptr, MSSA};
return LAM.getResult<LoopAccessAnalysis>(L, AR);
});
Index: llvm/lib/Transforms/Scalar/LoopDistribute.cpp
===================================================================
--- llvm/lib/Transforms/Scalar/LoopDistribute.cpp
+++ llvm/lib/Transforms/Scalar/LoopDistribute.cpp
@@ -1058,7 +1058,8 @@
auto &LAM = AM.getResult<LoopAnalysisManagerFunctionProxy>(F).getManager();
std::function<const LoopAccessInfo &(Loop &)> GetLAA =
[&](Loop &L) -> const LoopAccessInfo & {
- LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE, TLI, TTI, nullptr};
+ LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE,
+ TLI, TTI, nullptr, nullptr};
return LAM.getResult<LoopAccessAnalysis>(L, AR);
};
Index: llvm/lib/Analysis/BlockFrequencyInfo.cpp
===================================================================
--- llvm/lib/Analysis/BlockFrequencyInfo.cpp
+++ llvm/lib/Analysis/BlockFrequencyInfo.cpp
@@ -181,12 +181,36 @@
PAC.preservedSet<CFGAnalyses>());
}
+BlockFrequencyInfo::BFICallbackVH::BFICallbackVH(
+ std::shared_ptr<BlockFrequencyInfoImpl<BasicBlock>> BFI,
+ const BasicBlock *BB)
+ : CallbackVH(const_cast<BasicBlock *>(BB)), BFI(BFI), BB(BB) {}
+
+void BlockFrequencyInfo::BFICallbackVH::deleted() {
+ if (auto BFII = BFI.lock())
+ BFII->removeBlock(BB);
+ CallbackVH::deleted();
+}
+
+void BlockFrequencyInfo::BFICallbackVH::allUsesReplacedWith(Value *V) {
+ if (auto BFII = BFI.lock())
+ BFII->removeBlock(BB);
+ CallbackVH::allUsesReplacedWith(V);
+}
+
void BlockFrequencyInfo::calculate(const Function &F,
const BranchProbabilityInfo &BPI,
const LoopInfo &LI) {
if (!BFI)
BFI.reset(new ImplType);
BFI->calculate(F, BPI, LI);
+
+ BlockWatchers.clear();
+ auto RegisteredBlocks = BFI->getRegisteredBlocks();
+ BlockWatchers.reserve(RegisteredBlocks.size());
+ for (const auto *BB : RegisteredBlocks)
+ BlockWatchers.emplace_back(BFICallbackVH(BFI, BB));
+
if (ViewBlockFreqPropagationDAG != GVDT_None &&
(ViewBlockFreqFuncName.empty() ||
F.getName().equals(ViewBlockFreqFuncName))) {
Index: llvm/include/llvm/Transforms/Scalar/LoopPassManager.h
===================================================================
--- llvm/include/llvm/Transforms/Scalar/LoopPassManager.h
+++ llvm/include/llvm/Transforms/Scalar/LoopPassManager.h
@@ -41,6 +41,7 @@
#include "llvm/ADT/STLExtras.h"
#include "llvm/Analysis/AliasAnalysis.h"
#include "llvm/Analysis/BasicAliasAnalysis.h"
+#include "llvm/Analysis/BlockFrequencyInfo.h"
#include "llvm/Analysis/GlobalsModRef.h"
#include "llvm/Analysis/LoopAnalysisManager.h"
#include "llvm/Analysis/LoopInfo.h"
@@ -274,6 +275,7 @@
AM.getResult<ScalarEvolutionAnalysis>(F),
AM.getResult<TargetLibraryAnalysis>(F),
AM.getResult<TargetIRAnalysis>(F),
+ &AM.getResult<BlockFrequencyAnalysis>(F),
MSSA};
// Setup the loop analysis manager from its proxy. It is important that
@@ -370,6 +372,7 @@
PA.preserve<DominatorTreeAnalysis>();
PA.preserve<LoopAnalysis>();
PA.preserve<ScalarEvolutionAnalysis>();
+ PA.preserve<BlockFrequencyAnalysis>();
if (UseMemorySSA)
PA.preserve<MemorySSAAnalysis>();
// FIXME: What we really want to do here is preserve an AA category, but
Index: llvm/include/llvm/Analysis/LoopAnalysisManager.h
===================================================================
--- llvm/include/llvm/Analysis/LoopAnalysisManager.h
+++ llvm/include/llvm/Analysis/LoopAnalysisManager.h
@@ -57,6 +57,7 @@
ScalarEvolution &SE;
TargetLibraryInfo &TLI;
TargetTransformInfo &TTI;
+ BlockFrequencyInfo *BFI;
MemorySSA *MSSA;
};
Index: llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h
===================================================================
--- llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h
+++ llvm/include/llvm/Analysis/BlockFrequencyInfoImpl.h
@@ -847,10 +847,13 @@
/// (Running this until fixed point would "solve" the geometric
/// series by simulation.)
template <class BT> class BlockFrequencyInfoImpl : BlockFrequencyInfoImplBase {
+public:
+ using BlockT = typename bfi_detail::TypeMap<BT>::BlockT;
+
+private:
// This is part of a workaround for a GCC 4.7 crash on lambdas.
friend struct bfi_detail::BlockEdgesAdder<BT>;
- using BlockT = typename bfi_detail::TypeMap<BT>::BlockT;
using BlockKeyT = typename bfi_detail::TypeMap<BT>::BlockKeyT;
using FunctionT = typename bfi_detail::TypeMap<BT>::FunctionT;
using BranchProbabilityInfoT =
@@ -1015,6 +1018,13 @@
const BranchProbabilityInfoT &getBPI() const { return *BPI; }
+ /// Removes \p BB from internal datastructures.
+ /// \return false if the block did not exist.
+ bool removeBlock(const BlockT *BB);
+
+ /// \return all blocks known to this datastructure.
+ DenseSet<const BlockT *> getRegisteredBlocks() const;
+
/// Print the frequencies for the current function.
///
/// Prints the frequencies for the blocks in the current function.
@@ -1121,6 +1131,20 @@
}
}
+template <class BT>
+bool BlockFrequencyInfoImpl<BT>::removeBlock(const BlockT *BB) {
+ return Nodes.erase(BB);
+}
+
+template <class BT>
+DenseSet<const typename BlockFrequencyInfoImpl<BT>::BlockT *>
+BlockFrequencyInfoImpl<BT>::getRegisteredBlocks() const {
+ DenseSet<const BlockT *> Ret;
+ for (auto &BN : Nodes)
+ Ret.insert(BN.first);
+ return Ret;
+}
+
template <class BT> void BlockFrequencyInfoImpl<BT>::initializeRPOT() {
const BlockT *Entry = &F->front();
RPOT.reserve(F->size());
Index: llvm/include/llvm/Analysis/BlockFrequencyInfo.h
===================================================================
--- llvm/include/llvm/Analysis/BlockFrequencyInfo.h
+++ llvm/include/llvm/Analysis/BlockFrequencyInfo.h
@@ -15,10 +15,12 @@
#include "llvm/ADT/Optional.h"
#include "llvm/IR/PassManager.h"
+#include "llvm/IR/ValueHandle.h"
#include "llvm/Pass.h"
#include "llvm/Support/BlockFrequency.h"
#include <cstdint>
#include <memory>
+#include <vector>
namespace llvm {
@@ -37,7 +39,21 @@
class BlockFrequencyInfo {
using ImplType = BlockFrequencyInfoImpl<BasicBlock>;
- std::unique_ptr<ImplType> BFI;
+ class BFICallbackVH final : public CallbackVH {
+ std::weak_ptr<BlockFrequencyInfoImpl<BasicBlock>> BFI;
+ const BasicBlock *BB;
+
+ public:
+ BFICallbackVH(std::shared_ptr<BlockFrequencyInfoImpl<BasicBlock>> BFI,
+ const BasicBlock *BB);
+ virtual ~BFICallbackVH() = default;
+
+ virtual void deleted() override;
+ virtual void allUsesReplacedWith(Value *V) override;
+ };
+ std::vector<BFICallbackVH> BlockWatchers;
+
+ std::shared_ptr<ImplType> BFI;
public:
BlockFrequencyInfo();
Index: clang/test/CodeGen/thinlto-distributed-newpm.ll
===================================================================
--- clang/test/CodeGen/thinlto-distributed-newpm.ll
+++ clang/test/CodeGen/thinlto-distributed-newpm.ll
@@ -112,6 +112,9 @@
; CHECK-O: Running analysis: LoopAnalysis on main
; CHECK-O: Running pass: LCSSAPass on main
; CHECK-O: Finished {{.*}}Function pass manager run.
+; CHECK-O: Running analysis: BlockFrequencyAnalysis on main
+; CHECK-O: Running analysis: BranchProbabilityAnalysis on main
+; CHECK-O: Running analysis: PostDominatorTreeAnalysis on main
; CHECK-O: Running pass: SimplifyCFGPass on main
; CHECK-O: Running pass: InstCombinePass on main
; CHECK-O: Starting {{.*}}Function pass manager run.
@@ -136,7 +139,6 @@
; CHECK-O: Running pass: LCSSAPass on main
; CHECK-O: Finished {{.*}}Function pass manager run.
; CHECK-O: Running pass: ADCEPass on main
-; CHECK-O: Running analysis: PostDominatorTreeAnalysis on main
; CHECK-O: Running pass: SimplifyCFGPass on main
; CHECK-O: Running pass: InstCombinePass on main
; CHECK-O: Finished {{.*}}Function pass manager run.
@@ -146,10 +148,10 @@
; CHECK-O: Invalidating analysis: AAManager on main
; CHECK-O: Invalidating analysis: MemorySSAAnalysis on main
; CHECK-O: Invalidating analysis: LoopAnalysis on main
+; CHECK-O: Invalidating analysis: PostDominatorTreeAnalysis on main
; CHECK-O: Invalidating analysis: PhiValuesAnalysis on main
; CHECK-O: Invalidating analysis: MemoryDependenceAnalysis on main
; CHECK-O: Invalidating analysis: DemandedBitsAnalysis on main
-; CHECK-O: Invalidating analysis: PostDominatorTreeAnalysis on main
; CHECK-O: Invalidating analysis: CallGraphAnalysis
; CHECK-O: Finished {{.*}}Module pass manager run.
; CHECK-O: Starting {{.*}}Module pass manager run.
@@ -171,14 +173,14 @@
; CHECK-O: Running analysis: AAManager on main
; CHECK-O: Running analysis: BasicAA on main
; CHECK-O: Running analysis: ScalarEvolutionAnalysis on main
+; CHECK-O: Running analysis: BlockFrequencyAnalysis on main
+; CHECK-O: Running analysis: BranchProbabilityAnalysis on main
+; CHECK-O: Running analysis: PostDominatorTreeAnalysis on main
; CHECK-O: Running analysis: InnerAnalysisManagerProxy
; CHECK-O: Running pass: LoopRotatePass on Loop at depth 1 containing: %b
; CHECK-O: Running pass: LoopDistributePass on main
; CHECK-O: Running pass: InjectTLIMappings on main
; CHECK-O: Running pass: LoopVectorizePass on main
-; CHECK-O: Running analysis: BlockFrequencyAnalysis on main
-; CHECK-O: Running analysis: BranchProbabilityAnalysis on main
-; CHECK-O: Running analysis: PostDominatorTreeAnalysis on main
; CHECK-O: Running analysis: DemandedBitsAnalysis on main
; CHECK-O: Running pass: LoopLoadEliminationPass on main
; CHECK-O: Running analysis: LoopAccessAnalysis on Loop at depth 1 containing: %b
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
