tejohnson created this revision.
tejohnson added reviewers: pcc, evgeny777.
Herald added subscribers: arphaman, dexonsmith, steven_wu, hiraditya, Prazek,
mehdi_amini.
Herald added projects: clang, LLVM.
Currently type test assume sequences inserted for devirtualization are
removed during WPD. This patch delays their removal until later in the
optimization pipeline. This is an enabler for upcoming enhancements to
indirect call promotion, for example streamlined promotion guard
sequences that compare against vtable address instead of the target
function, when there are small number of possible vtables (either
determined via WPD or by in-progress type profiling). We need the type
tests to correlate the callsites with the address point offset needed in
the compare sequence, and optionally to associated type summary info
computed during WPD.
This depends on work in D71913 <https://reviews.llvm.org/D71913> to enable
invocation of LowerTypeTests to
drop type test assume sequences, which will now be invoked following ICP
in the ThinLTO post-LTO link pipelines, and also after the existing
export phase LowerTypeTests invocation in regular LTO (which is already
after ICP). We cannot simply move the existing import phase
LowerTypeTests pass later in the ThinLTO post link pipelines, as the
comment in PassBuilder.cpp notes (it must run early because when
performing CFI other passes may disturb the sequences it looks for).
This necessitated adding a new type test resolution "Unknown" that we
can use on the type test assume sequences previously removed by WPD,
that we now want LTT to ignore.
Depends on D71913 <https://reviews.llvm.org/D71913>.
Repository:
rG LLVM Github Monorepo
https://reviews.llvm.org/D73242
Files:
clang/test/CodeGen/thinlto-distributed-cfi-devirt.ll
clang/test/CodeGen/thinlto-distributed-cfi.ll
llvm/include/llvm/IR/ModuleSummaryIndex.h
llvm/include/llvm/IR/ModuleSummaryIndexYAML.h
llvm/lib/AsmParser/LLParser.cpp
llvm/lib/IR/AsmWriter.cpp
llvm/lib/Passes/PassBuilder.cpp
llvm/lib/Transforms/IPO/LowerTypeTests.cpp
llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp
llvm/test/Other/new-pm-lto-defaults.ll
llvm/test/Other/new-pm-thinlto-defaults.ll
llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll
llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll
llvm/test/ThinLTO/X86/cfi-icall-only-defuse.ll
llvm/test/ThinLTO/X86/cfi-icall.ll
llvm/test/Transforms/WholeProgramDevirt/branch-funnel.ll
llvm/test/Transforms/WholeProgramDevirt/export-single-impl.ll
llvm/test/Transforms/WholeProgramDevirt/export-uniform-ret-val.ll
llvm/test/Transforms/WholeProgramDevirt/export-unique-ret-val.ll
llvm/test/Transforms/WholeProgramDevirt/export-vcp.ll
llvm/test/Transforms/WholeProgramDevirt/import-indir.ll
llvm/test/Transforms/WholeProgramDevirt/uniform-retval.ll
Index: llvm/test/Transforms/WholeProgramDevirt/uniform-retval.ll
===================================================================
--- llvm/test/Transforms/WholeProgramDevirt/uniform-retval.ll
+++ llvm/test/Transforms/WholeProgramDevirt/uniform-retval.ll
@@ -25,7 +25,7 @@
%fptr = load i8*, i8** %fptrptr
%fptr_casted = bitcast i8* %fptr to i32 (i8*)*
%result = call i32 %fptr_casted(i8* %obj)
- ; CHECK-NOT: call
+ ; CHECK-NOT: call i32 %
; CHECK: ret i32 123
ret i32 %result
}
Index: llvm/test/Transforms/WholeProgramDevirt/import-indir.ll
===================================================================
--- llvm/test/Transforms/WholeProgramDevirt/import-indir.ll
+++ llvm/test/Transforms/WholeProgramDevirt/import-indir.ll
@@ -32,7 +32,7 @@
; SUMMARY-NEXT: TypeIdMap:
; SUMMARY-NEXT: typeid1:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
Index: llvm/test/Transforms/WholeProgramDevirt/export-vcp.ll
===================================================================
--- llvm/test/Transforms/WholeProgramDevirt/export-vcp.ll
+++ llvm/test/Transforms/WholeProgramDevirt/export-vcp.ll
@@ -9,7 +9,7 @@
; SUMMARY: TypeIdMap:
; SUMMARY-NEXT: typeid3:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
@@ -29,7 +29,7 @@
; SUMMARY-ARM-NEXT: Bit: 1
; SUMMARY-NEXT: typeid4:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
Index: llvm/test/Transforms/WholeProgramDevirt/export-unique-ret-val.ll
===================================================================
--- llvm/test/Transforms/WholeProgramDevirt/export-unique-ret-val.ll
+++ llvm/test/Transforms/WholeProgramDevirt/export-unique-ret-val.ll
@@ -6,7 +6,7 @@
; SUMMARY: TypeIdMap:
; SUMMARY-NEXT: typeid3:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
@@ -24,7 +24,7 @@
; SUMMARY-NEXT: Bit: 0
; SUMMARY-NEXT: typeid4:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
Index: llvm/test/Transforms/WholeProgramDevirt/export-uniform-ret-val.ll
===================================================================
--- llvm/test/Transforms/WholeProgramDevirt/export-uniform-ret-val.ll
+++ llvm/test/Transforms/WholeProgramDevirt/export-uniform-ret-val.ll
@@ -6,7 +6,7 @@
; SUMMARY: TypeIdMap:
; SUMMARY-NEXT: typeid4:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
Index: llvm/test/Transforms/WholeProgramDevirt/export-single-impl.ll
===================================================================
--- llvm/test/Transforms/WholeProgramDevirt/export-single-impl.ll
+++ llvm/test/Transforms/WholeProgramDevirt/export-single-impl.ll
@@ -4,7 +4,7 @@
; SUMMARY: TypeIdMap:
; SUMMARY-NEXT: typeid3:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
@@ -17,7 +17,7 @@
; SUMMARY-NEXT: ResByArg:
; SUMMARY-NEXT: typeid1:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
@@ -30,7 +30,7 @@
; SUMMARY-NEXT: ResByArg:
; SUMMARY-NEXT: typeid2:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
@@ -43,7 +43,7 @@
; SUMMARY-NEXT: ResByArg:
; SUMMARY-NEXT: typeid4:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
Index: llvm/test/Transforms/WholeProgramDevirt/branch-funnel.ll
===================================================================
--- llvm/test/Transforms/WholeProgramDevirt/branch-funnel.ll
+++ llvm/test/Transforms/WholeProgramDevirt/branch-funnel.ll
@@ -10,7 +10,7 @@
; SUMMARY: TypeIdMap:
; SUMMARY-NEXT: typeid3:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
@@ -23,7 +23,7 @@
; SUMMARY-NEXT: ResByArg:
; SUMMARY-NEXT: typeid1:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
@@ -36,7 +36,7 @@
; SUMMARY-NEXT: ResByArg:
; SUMMARY-NEXT: typeid2:
; SUMMARY-NEXT: TTRes:
-; SUMMARY-NEXT: Kind: Unsat
+; SUMMARY-NEXT: Kind: Unknown
; SUMMARY-NEXT: SizeM1BitWidth: 0
; SUMMARY-NEXT: AlignLog2: 0
; SUMMARY-NEXT: SizeM1: 0
Index: llvm/test/ThinLTO/X86/cfi-icall.ll
===================================================================
--- llvm/test/ThinLTO/X86/cfi-icall.ll
+++ llvm/test/ThinLTO/X86/cfi-icall.ll
@@ -29,7 +29,7 @@
; COMBINED: <GLOBALVAL_SUMMARY_BLOCK
; COMBINED: <CFI_FUNCTION_DEFS op0=0 op1=3/>
; COMBINED: <CFI_FUNCTION_DECLS op0=3 op1=3/>
-; COMBINED: <TYPE_ID op0=6 op1=7 op2=4 op3=7 op4=0 op5=0 op6=0 op7=0/>
+; COMBINED: <TYPE_ID op0=6 op1=7 op2=5 op3=7 op4=0 op5=0 op6=0 op7=0/>
; COMBINED: </GLOBALVAL_SUMMARY_BLOCK>
; COMBINED: <STRTAB_BLOCK
Index: llvm/test/ThinLTO/X86/cfi-icall-only-defuse.ll
===================================================================
--- llvm/test/ThinLTO/X86/cfi-icall-only-defuse.ll
+++ llvm/test/ThinLTO/X86/cfi-icall-only-defuse.ll
@@ -41,7 +41,7 @@
; FOOBAZ: <GLOBALVAL_SUMMARY_BLOCK
; FOOBAZ: <CFI_FUNCTION_DEFS op0=0 op1=3 op2=3 op3=3 op4=6 op5=3/>
-; FOOBAZ: <TYPE_ID op0=9 op1=2 op2=4 op3=7 op4=0 op5=0 op6=0 op7=0/>
+; FOOBAZ: <TYPE_ID op0=9 op1=2 op2=5 op3=7 op4=0 op5=0 op6=0 op7=0/>
; FOOBAZ: </GLOBALVAL_SUMMARY_BLOCK>
; FOOBAZ: <STRTAB_BLOCK
; FOOBAZ-NEXT: <BLOB abbrevid=4/> blob data = 'barbazfoot1'
@@ -49,7 +49,7 @@
; BARQUX: <GLOBALVAL_SUMMARY_BLOCK
; BARQUX: <CFI_FUNCTION_DEFS op0=0 op1=3 op2=3 op3=3 op4=6 op5=3/>
-; BARQUX: <TYPE_ID op0=9 op1=2 op2=4 op3=7 op4=0 op5=0 op6=0 op7=0/>
+; BARQUX: <TYPE_ID op0=9 op1=2 op2=5 op3=7 op4=0 op5=0 op6=0 op7=0/>
; BARQUX: </GLOBALVAL_SUMMARY_BLOCK>
; BARQUX: <STRTAB_BLOCK
; BARQUX-NEXT: <BLOB abbrevid=4/> blob data = 'barbazquxt1'
Index: llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll
===================================================================
--- llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll
+++ llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll
@@ -59,6 +59,7 @@
; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
; CHECK-O-NEXT: Running pass: PGOIndirectCallPromotion
+; CHECK-O-NEXT: Running pass: LowerTypeTestsPass
; CHECK-O-NEXT: Running pass: IPSCCPPass
; CHECK-O-NEXT: Running pass: CalledValuePropagationPass
; CHECK-O-NEXT: Running pass: GlobalOptPass
Index: llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll
===================================================================
--- llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll
+++ llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll
@@ -48,6 +48,7 @@
; CHECK-O-NEXT: Running pass: LowerExpectIntrinsicPass
; CHECK-O3-NEXT: Running pass: CallSiteSplittingPass
; CHECK-O-NEXT: Finished {{.*}}Function pass manager run.
+; CHECK-O-NEXT: Running pass: LowerTypeTestsPass
; CHECK-O-NEXT: Running pass: IPSCCPPass
; CHECK-O-NEXT: Running pass: CalledValuePropagationPass
; CHECK-O-NEXT: Running pass: GlobalOptPass
Index: llvm/test/Other/new-pm-thinlto-defaults.ll
===================================================================
--- llvm/test/Other/new-pm-thinlto-defaults.ll
+++ llvm/test/Other/new-pm-thinlto-defaults.ll
@@ -79,6 +79,7 @@
; CHECK-O-NEXT: Running pass: LowerExpectIntrinsicPass
; CHECK-O3-NEXT: Running pass: CallSiteSplittingPass
; CHECK-O-NEXT: Finished llvm::Function pass manager run.
+; CHECK-POSTLINK-O-NEXT: Running pass: LowerTypeTestsPass
; CHECK-O-NEXT: Running pass: IPSCCPPass
; CHECK-O-NEXT: Running pass: CalledValuePropagationPass
; CHECK-O-NEXT: Running pass: GlobalOptPass
Index: llvm/test/Other/new-pm-lto-defaults.ll
===================================================================
--- llvm/test/Other/new-pm-lto-defaults.ll
+++ llvm/test/Other/new-pm-lto-defaults.ll
@@ -92,6 +92,7 @@
; CHECK-O2-NEXT: Running analysis: DemandedBitsAnalysis
; CHECK-O2-NEXT: Running pass: CrossDSOCFIPass
; CHECK-O2-NEXT: Running pass: LowerTypeTestsPass
+; CHECK-O-NEXT: Running pass: LowerTypeTestsPass
; CHECK-O2-NEXT: Running pass: ModuleToFunctionPassAdaptor<{{.*}}SimplifyCFGPass>
; CHECK-O2-NEXT: Running pass: EliminateAvailableExternallyPass
; CHECK-O2-NEXT: Running pass: GlobalDCEPass
Index: llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp
===================================================================
--- llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp
+++ llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp
@@ -504,7 +504,9 @@
bool areRemarksEnabled();
- void scanTypeTestUsers(Function *TypeTestFunc);
+ void
+ scanTypeTestUsers(Function *TypeTestFunc,
+ DenseMap<Metadata *, std::set<TypeMemberInfo>> &TypeIdMap);
void scanTypeCheckedLoadUsers(Function *TypeCheckedLoadFunc);
void buildTypeIdentifierMap(
@@ -1631,7 +1633,9 @@
return false;
}
-void DevirtModule::scanTypeTestUsers(Function *TypeTestFunc) {
+void DevirtModule::scanTypeTestUsers(
+ Function *TypeTestFunc,
+ DenseMap<Metadata *, std::set<TypeMemberInfo>> &TypeIdMap) {
// Find all virtual calls via a virtual table pointer %p under an assumption
// of the form llvm.assume(llvm.type.test(%p, %md)). This indicates that %p
// points to a member of the type identifier %md. Group calls by (type ID,
@@ -1651,10 +1655,10 @@
auto &DT = LookupDomTree(*CI->getFunction());
findDevirtualizableCallsForTypeTest(DevirtCalls, Assumes, CI, DT);
+ Metadata *TypeId =
+ cast<MetadataAsValue>(CI->getArgOperand(1))->getMetadata();
// If we found any, add them to CallSlots.
if (!Assumes.empty()) {
- Metadata *TypeId =
- cast<MetadataAsValue>(CI->getArgOperand(1))->getMetadata();
Value *Ptr = CI->getArgOperand(0)->stripPointerCasts();
for (DevirtCallSite Call : DevirtCalls) {
// Only add this CallSite if we haven't seen it before. The vtable
@@ -1667,6 +1671,13 @@
}
}
+ // If we have any uses on type metadata, keep the type test assumes for
+ // later analysis. Otherwise remove as they aren't useful, and
+ // LowerTypeTests will think they are Unsat and lower to False, which
+ // breaks any uses on assumes.
+ if (TypeIdMap.count(TypeId))
+ continue;
+
// We no longer need the assumes or the type test.
for (auto Assume : Assumes)
Assume->eraseFromParent();
@@ -1865,8 +1876,13 @@
(!TypeCheckedLoadFunc || TypeCheckedLoadFunc->use_empty()))
return false;
+ // Rebuild type metadata into a map for easy lookup.
+ std::vector<VTableBits> Bits;
+ DenseMap<Metadata *, std::set<TypeMemberInfo>> TypeIdMap;
+ buildTypeIdentifierMap(Bits, TypeIdMap);
+
if (TypeTestFunc && AssumeFunc)
- scanTypeTestUsers(TypeTestFunc);
+ scanTypeTestUsers(TypeTestFunc, TypeIdMap);
if (TypeCheckedLoadFunc)
scanTypeCheckedLoadUsers(TypeCheckedLoadFunc);
@@ -1888,10 +1904,6 @@
return true;
}
- // Rebuild type metadata into a map for easy lookup.
- std::vector<VTableBits> Bits;
- DenseMap<Metadata *, std::set<TypeMemberInfo>> TypeIdMap;
- buildTypeIdentifierMap(Bits, TypeIdMap);
if (TypeIdMap.empty())
return true;
@@ -1948,14 +1960,17 @@
// function implementation at offset S.first.ByteOffset, and add to
// TargetsForSlot.
std::vector<VirtualCallTarget> TargetsForSlot;
+ WholeProgramDevirtResolution *Res = nullptr;
+ if (ExportSummary && isa<MDString>(S.first.TypeID))
+ // Create the type id summary resolution regardlness of whether we can
+ // devirtualize, so that lower type tests knows the type id is used on
+ // a global and not Unsat.
+ Res = &ExportSummary
+ ->getOrInsertTypeIdSummary(
+ cast<MDString>(S.first.TypeID)->getString())
+ .WPDRes[S.first.ByteOffset];
if (tryFindVirtualCallTargets(TargetsForSlot, TypeIdMap[S.first.TypeID],
S.first.ByteOffset)) {
- WholeProgramDevirtResolution *Res = nullptr;
- if (ExportSummary && isa<MDString>(S.first.TypeID))
- Res = &ExportSummary
- ->getOrInsertTypeIdSummary(
- cast<MDString>(S.first.TypeID)->getString())
- .WPDRes[S.first.ByteOffset];
if (!trySingleImplDevirt(ExportSummary, TargetsForSlot, S.second, Res)) {
DidVirtualConstProp |=
@@ -2069,11 +2084,14 @@
std::vector<ValueInfo> TargetsForSlot;
auto TidSummary = ExportSummary.getTypeIdCompatibleVtableSummary(S.first.TypeID);
assert(TidSummary);
+ // Create the type id summary resolution regardlness of whether we can
+ // devirtualize, so that lower type tests knows the type id is used on
+ // a global and not Unsat.
+ WholeProgramDevirtResolution *Res =
+ &ExportSummary.getOrInsertTypeIdSummary(S.first.TypeID)
+ .WPDRes[S.first.ByteOffset];
if (tryFindVirtualCallTargets(TargetsForSlot, *TidSummary,
S.first.ByteOffset)) {
- WholeProgramDevirtResolution *Res =
- &ExportSummary.getOrInsertTypeIdSummary(S.first.TypeID)
- .WPDRes[S.first.ByteOffset];
if (!trySingleImplDevirt(TargetsForSlot, S.first, S.second, Res,
DevirtTargets))
Index: llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
===================================================================
--- llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
+++ llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
@@ -478,6 +478,7 @@
MPM.add(createBarrierNoopPass());
if (PerformThinLTO) {
+ MPM.add(createLowerTypeTestsPass(nullptr, nullptr, true));
// Drop available_externally and unreferenced globals. This is necessary
// with ThinLTO in order to avoid leaving undefined references to dead
// globals in the object file.
@@ -511,9 +512,11 @@
// inter-module indirect calls. For that we perform indirect call promotion
// earlier in the pass pipeline, here before globalopt. Otherwise imported
// available_externally functions look unreferenced and are removed.
- if (PerformThinLTO)
+ if (PerformThinLTO) {
MPM.add(createPGOIndirectCallPromotionLegacyPass(/*InLTO = */ true,
!PGOSampleUse.empty()));
+ MPM.add(createLowerTypeTestsPass(nullptr, nullptr, true));
+ }
// For SamplePGO in ThinLTO compile phase, we do not want to unroll loops
// as it will change the CFG too much to make the 2nd profile annotation
@@ -1013,8 +1016,8 @@
PM.add(createVerifierPass());
if (ImportSummary) {
- // These passes import type identifier resolutions for whole-program
- // devirtualization and CFI. They must run early because other passes may
+ // This pass imports type identifier resolutions for whole-program
+ // devirtualization and CFI. It must run early because other passes may
// disturb the specific instruction patterns that these passes look for,
// creating dependencies on resolutions that may not appear in the summary.
//
@@ -1062,6 +1065,9 @@
// control flow integrity mechanisms (-fsanitize=cfi*) and needs to run at
// link time if CFI is enabled. The pass does nothing if CFI is disabled.
PM.add(createLowerTypeTestsPass(ExportSummary, nullptr));
+ // Run a second time to clean up any type tests left behind by WPD for use
+ // in ICP (which is performed earlier than this in the regular LTO pipeline).
+ PM.add(createLowerTypeTestsPass(nullptr, nullptr, true));
if (OptLevel != 0)
addLateLTOOptimizationPasses(PM);
Index: llvm/lib/Transforms/IPO/LowerTypeTests.cpp
===================================================================
--- llvm/lib/Transforms/IPO/LowerTypeTests.cpp
+++ llvm/lib/Transforms/IPO/LowerTypeTests.cpp
@@ -735,6 +735,9 @@
/// replace the call with.
Value *LowerTypeTestsModule::lowerTypeTestCall(Metadata *TypeId, CallInst *CI,
const TypeIdLowering &TIL) {
+ // Delay lowering if the resolution is currently unknown.
+ if (TIL.TheKind == TypeTestResolution::Unknown)
+ return nullptr;
if (TIL.TheKind == TypeTestResolution::Unsat)
return ConstantInt::getFalse(M.getContext());
@@ -1043,8 +1046,10 @@
TypeIdLowering TIL = importTypeId(TypeIdStr->getString());
Value *Lowered = lowerTypeTestCall(TypeIdStr, CI, TIL);
- CI->replaceAllUsesWith(Lowered);
- CI->eraseFromParent();
+ if (Lowered) {
+ CI->replaceAllUsesWith(Lowered);
+ CI->eraseFromParent();
+ }
}
// ThinLTO backend: the function F has a jump table entry; update this module
@@ -1167,8 +1172,10 @@
for (CallInst *CI : TIUI.CallSites) {
++NumTypeTestCallsLowered;
Value *Lowered = lowerTypeTestCall(TypeId, CI, TIL);
- CI->replaceAllUsesWith(Lowered);
- CI->eraseFromParent();
+ if (Lowered) {
+ CI->replaceAllUsesWith(Lowered);
+ CI->eraseFromParent();
+ }
}
}
}
Index: llvm/lib/Passes/PassBuilder.cpp
===================================================================
--- llvm/lib/Passes/PassBuilder.cpp
+++ llvm/lib/Passes/PassBuilder.cpp
@@ -747,6 +747,12 @@
true /* SamplePGO */));
}
+ // Lower type metadata and the type.test intrinsic in the ThinLTO
+ // post link pipeline after ICP. This is to enable usage of the type
+ // tests in ICP sequences.
+ if (Phase == ThinLTOPhase::PostLink)
+ MPM.addPass(LowerTypeTestsPass(nullptr, nullptr, true));
+
// Interprocedural constant propagation now that basic cleanup has occurred
// and prior to optimizing globals.
// FIXME: This position in the pipeline hasn't been carefully considered in
@@ -1169,6 +1175,9 @@
// metadata and intrinsics.
MPM.addPass(WholeProgramDevirtPass(ExportSummary, nullptr));
MPM.addPass(LowerTypeTestsPass(ExportSummary, nullptr));
+ // Run a second time to clean up any type tests left behind by WPD for use
+ // in ICP.
+ MPM.addPass(LowerTypeTestsPass(nullptr, nullptr, true));
return MPM;
}
@@ -1235,6 +1244,10 @@
// The LowerTypeTestsPass needs to run to lower type metadata and the
// type.test intrinsics. The pass does nothing if CFI is disabled.
MPM.addPass(LowerTypeTestsPass(ExportSummary, nullptr));
+ // Run a second time to clean up any type tests left behind by WPD for use
+ // in ICP (which is performed earlier than this in the regular LTO
+ // pipeline).
+ MPM.addPass(LowerTypeTestsPass(nullptr, nullptr, true));
return MPM;
}
@@ -1362,6 +1375,9 @@
// to be run at link time if CFI is enabled. This pass does nothing if
// CFI is disabled.
MPM.addPass(LowerTypeTestsPass(ExportSummary, nullptr));
+ // Run a second time to clean up any type tests left behind by WPD for use
+ // in ICP (which is performed earlier than this in the regular LTO pipeline).
+ MPM.addPass(LowerTypeTestsPass(nullptr, nullptr, true));
// Enable splitting late in the FullLTO post-link pipeline. This is done in
// the same stage in the old pass manager (\ref addLateLTOOptimizationPasses).
Index: llvm/lib/IR/AsmWriter.cpp
===================================================================
--- llvm/lib/IR/AsmWriter.cpp
+++ llvm/lib/IR/AsmWriter.cpp
@@ -2769,6 +2769,8 @@
static const char *getTTResKindName(TypeTestResolution::Kind K) {
switch (K) {
+ case TypeTestResolution::Unknown:
+ return "unknown";
case TypeTestResolution::Unsat:
return "unsat";
case TypeTestResolution::ByteArray:
Index: llvm/lib/AsmParser/LLParser.cpp
===================================================================
--- llvm/lib/AsmParser/LLParser.cpp
+++ llvm/lib/AsmParser/LLParser.cpp
@@ -7659,6 +7659,9 @@
return true;
switch (Lex.getKind()) {
+ case lltok::kw_unknown:
+ TTRes.TheKind = TypeTestResolution::Unknown;
+ break;
case lltok::kw_unsat:
TTRes.TheKind = TypeTestResolution::Unsat;
break;
Index: llvm/include/llvm/IR/ModuleSummaryIndexYAML.h
===================================================================
--- llvm/include/llvm/IR/ModuleSummaryIndexYAML.h
+++ llvm/include/llvm/IR/ModuleSummaryIndexYAML.h
@@ -17,6 +17,7 @@
template <> struct ScalarEnumerationTraits<TypeTestResolution::Kind> {
static void enumeration(IO &io, TypeTestResolution::Kind &value) {
+ io.enumCase(value, "Unknown", TypeTestResolution::Unknown);
io.enumCase(value, "Unsat", TypeTestResolution::Unsat);
io.enumCase(value, "ByteArray", TypeTestResolution::ByteArray);
io.enumCase(value, "Inline", TypeTestResolution::Inline);
Index: llvm/include/llvm/IR/ModuleSummaryIndex.h
===================================================================
--- llvm/include/llvm/IR/ModuleSummaryIndex.h
+++ llvm/include/llvm/IR/ModuleSummaryIndex.h
@@ -827,13 +827,14 @@
/// details on each kind of check; the enumerators are described with
/// reference to that document.
enum Kind {
+ Unknown, ///< Unknown (analysis not performed, don't lower)
Unsat, ///< Unsatisfiable type (i.e. no global has this type metadata)
ByteArray, ///< Test a byte array (first example)
Inline, ///< Inlined bit vector ("Short Inline Bit Vectors")
Single, ///< Single element (last example in "Short Inline Bit Vectors")
AllOnes, ///< All-ones bit vector ("Eliminating Bit Vector Checks for
/// All-Ones Bit Vectors")
- } TheKind = Unsat;
+ } TheKind = Unknown;
/// Range of size-1 expressed as a bit width. For example, if the size is in
/// range [1,256], this number will be 8. This helps generate the most compact
Index: clang/test/CodeGen/thinlto-distributed-cfi.ll
===================================================================
--- clang/test/CodeGen/thinlto-distributed-cfi.ll
+++ clang/test/CodeGen/thinlto-distributed-cfi.ll
@@ -14,7 +14,7 @@
; Check that typeids are in the index.
; RUN: llvm-bcanalyzer -dump %t.o.thinlto.bc | FileCheck %s
; CHECK-LABEL: <GLOBALVAL_SUMMARY_BLOCK
-; CHECK: <TYPE_ID op0=0 op1=6 op2=3 op3=0 op4=0 op5=0 op6=0 op7=0/>
+; CHECK: <TYPE_ID op0=0 op1=6 op2=4 op3=0 op4=0 op5=0 op6=0 op7=0/>
; CHECK-LABEL: </GLOBALVAL_SUMMARY_BLOCK
; CHECK-LABEL: <STRTAB_BLOCK
; CHECK: blob data = '_ZTS1A'
Index: clang/test/CodeGen/thinlto-distributed-cfi-devirt.ll
===================================================================
--- clang/test/CodeGen/thinlto-distributed-cfi-devirt.ll
+++ clang/test/CodeGen/thinlto-distributed-cfi-devirt.ll
@@ -26,7 +26,7 @@
; Ensure that typeids are in the index.
; RUN: llvm-bcanalyzer -dump %t.o.thinlto.bc | FileCheck %s
; CHECK-LABEL: <GLOBALVAL_SUMMARY_BLOCK
-; CHECK: <TYPE_ID op0=0 op1=6 op2=4 op3=7 op4=0 op5=0 op6=0 op7=0 op8=0 op9=2 op10=6 op11=0 op12=0 op13=8 op14=1 op15=6 op16=9 op17=0/>
+; CHECK: <TYPE_ID op0=0 op1=6 op2=5 op3=7 op4=0 op5=0 op6=0 op7=0 op8=0 op9=2 op10=6 op11=0 op12=0 op13=8 op14=1 op15=6 op16=9 op17=0/>
; CHECK-LABEL: </GLOBALVAL_SUMMARY_BLOCK
; CHECK-LABEL: <STRTAB_BLOCK
; CHECK: blob data = '_ZTS1A_ZN1A1nEi'
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
