https://github.com/4ast updated https://github.com/llvm/llvm-project/pull/233140
>From 0194dca2b8ac1b0345146dfc3bc014dedca97722 Mon Sep 17 00:00:00 2001 From: Alexei Starovoitov <[email protected]> Date: Sat, 17 Oct 2026 22:21:21 +0000 Subject: [PATCH] [BPF] Lower unwinding to unwind_goto and unwind_exit Replace the .bpf_cleanup table with two new instructions, so that neither a loader nor the kernel has to dispatch landing pads. unwind_goto pad BPF_JMP | BPF_JCOND, src_reg = 1 unwind_exit BPF_JMP | BPF_EXIT, src_reg = 1 'call f; unwind_goto pad' goes to the pad when f left with unwind_exit and falls through when it left with exit. No register says which one it was: how one instruction tells the other is up to the JIT. In everything else unwind_exit is exit. ret exit invoke f() unwind pad call f; unwind_goto pad resume r0 = 0; unwind_exit call _Unwind_RaiseException r0 = 0; unwind_exit invoke _Unwind_RaiseException goto pad A call that may unwind and has no landing pad gets one that only resumes. The first version of this change kept the bit in r3: 'r3 = 0; exit', 'r3 = 1; exit' and 'if r3 != 0 goto pad'. r3 is an ordinary register and an argument, and the kernel JITs had to recognize the pattern by its shape to keep the branch off the normal path. With the instructions there is nothing on the normal path to begin with: 'ret' is 'exit' again, the BTF decl_tag that told the kernel which functions set r3 is gone, and a nounwind function whose address is taken does not have to know that its caller looks. invoke, landingpad and resume only carry an unwind on. Something has to start it. With the Itanium ABI a language runtime calls _Unwind_RaiseException() for that, e.g. at the end of a panic handler. On other targets it is a function of the unwinder library. Here the backend lowers the call, unless the program defines the function. To start an unwind and to carry one on is the same thing: leave the frame with unwind_exit, after its own pad if it has one. By its prototype the function comes back when it fails. Here it cannot fail, so what follows the call is dropped. resume is left to DwarfEHPrepare, which turns it into a call of _Unwind_Resume() like on other targets. The backend takes that name from the libcall table. Neither function reaches the object file. Not being nounwind is the default in IR, so that alone does not opt a function in. It also needs uwtable, which rustc sets on every function with -C panic=unwind. clang needs -fexceptions -funwind-tables. A landing pad in a function without uwtable is an error. So is the start of an unwind in a function that cannot unwind: there is no unwind_goto after the calls of it. Every pad is entered whatever was thrown, so only catch-all pads are supported: cleanup, catch of a null type, and an empty filter, which is how rustc guards a function that must not unwind. A pad expects the exception object in r0. There is none, so a pad that looks at it starts with r0 = 0. unwind_exit returns r0 like exit does, hence 'r0 = 0' in front of it. The kernel wants unwind_goto right after the call. A callee that does not return needs it as well: that is how its unwind_exit finds the pad. unwind_goto has a 16-bit offset. BPFMIPreEmitPeephole::adjustBranch() took the third operand of every conditional branch for its target. Now it takes the last one and retargets the branch in place. unwind_goto is inserted after register allocation. Until then the edge from the call to the pad is an EH edge, which keeps only r6-r9 and the stack alive over it. That is what a return of the callee preserves. clang defines __BPF_FEATURE_UNWIND. Add MachineFunction::clearLandingPads(), since the pads become ordinary blocks and there is no exception table to emit. --- clang/lib/Basic/Targets/BPF.cpp | 1 + .../test/Preprocessor/bpf-predefined-macros.c | 8 + llvm/include/llvm/CodeGen/MachineFunction.h | 4 + .../lib/Target/BPF/AsmParser/BPFAsmParser.cpp | 2 + llvm/lib/Target/BPF/BPF.h | 2 + llvm/lib/Target/BPF/BPFAsmPrinter.cpp | 55 ---- llvm/lib/Target/BPF/BPFAsmPrinter.h | 1 - llvm/lib/Target/BPF/BPFInstrInfo.td | 16 +- llvm/lib/Target/BPF/BPFMIPeephole.cpp | 31 +- llvm/lib/Target/BPF/BPFTargetMachine.cpp | 2 + llvm/lib/Target/BPF/BPFUnwind.cpp | 297 +++++++++++++++++ llvm/lib/Target/BPF/CMakeLists.txt | 1 + .../BPF/MCTargetDesc/BPFMCTargetDesc.cpp | 3 +- llvm/test/CodeGen/BPF/cleanup-section.ll | 160 ---------- llvm/test/CodeGen/BPF/unwind-far-pad.ll | 36 +++ llvm/test/CodeGen/BPF/unwind-own-raise.ll | 20 ++ llvm/test/CodeGen/BPF/unwind-reject-filter.ll | 24 ++ .../CodeGen/BPF/unwind-reject-no-uwtable.ll | 22 ++ .../CodeGen/BPF/unwind-reject-nounwind.ll | 14 + ...-catch.ll => unwind-reject-typed-catch.ll} | 4 +- llvm/test/CodeGen/BPF/unwind.ll | 298 ++++++++++++++++++ llvm/test/MC/BPF/insn-unwind.s | 23 ++ 22 files changed, 776 insertions(+), 248 deletions(-) create mode 100644 llvm/lib/Target/BPF/BPFUnwind.cpp delete mode 100644 llvm/test/CodeGen/BPF/cleanup-section.ll create mode 100644 llvm/test/CodeGen/BPF/unwind-far-pad.ll create mode 100644 llvm/test/CodeGen/BPF/unwind-own-raise.ll create mode 100644 llvm/test/CodeGen/BPF/unwind-reject-filter.ll create mode 100644 llvm/test/CodeGen/BPF/unwind-reject-no-uwtable.ll create mode 100644 llvm/test/CodeGen/BPF/unwind-reject-nounwind.ll rename llvm/test/CodeGen/BPF/{cleanup-reject-typed-catch.ll => unwind-reject-typed-catch.ll} (80%) create mode 100644 llvm/test/CodeGen/BPF/unwind.ll create mode 100644 llvm/test/MC/BPF/insn-unwind.s diff --git a/clang/lib/Basic/Targets/BPF.cpp b/clang/lib/Basic/Targets/BPF.cpp index 100769ea4cdb13..29e4c86ad88621 100644 --- a/clang/lib/Basic/Targets/BPF.cpp +++ b/clang/lib/Basic/Targets/BPF.cpp @@ -47,6 +47,7 @@ void BPFTargetInfo::getTargetDefines(const LangOptions &Opts, Builder.defineMacro("__BPF_FEATURE_MAY_GOTO"); Builder.defineMacro("__BPF_FEATURE_ATOMIC_MEM_ORDERING"); Builder.defineMacro("__BPF_FEATURE_STACK_ARGUMENT"); + Builder.defineMacro("__BPF_FEATURE_UNWIND"); if (CPU.empty()) CPU = "v3"; diff --git a/clang/test/Preprocessor/bpf-predefined-macros.c b/clang/test/Preprocessor/bpf-predefined-macros.c index 90287b7b24e955..d3d59a979e06bf 100644 --- a/clang/test/Preprocessor/bpf-predefined-macros.c +++ b/clang/test/Preprocessor/bpf-predefined-macros.c @@ -79,6 +79,9 @@ int x; #ifdef __BPF_FEATURE_STACK_ARGUMENT int y; #endif +#ifdef __BPF_FEATURE_UNWIND +int z; +#endif // CHECK: int b; // CHECK: int c; @@ -131,6 +134,11 @@ int y; // CPU_V3: int y; // CPU_V4: int y; +// CPU_V1: int z; +// CPU_V2: int z; +// CPU_V3: int z; +// CPU_V4: int z; + // CPU_GENERIC: int g; // CPU_PROBE: int f; diff --git a/llvm/include/llvm/CodeGen/MachineFunction.h b/llvm/include/llvm/CodeGen/MachineFunction.h index 7a6f71b5648a64..6387ad0a90d67a 100644 --- a/llvm/include/llvm/CodeGen/MachineFunction.h +++ b/llvm/include/llvm/CodeGen/MachineFunction.h @@ -1340,6 +1340,10 @@ class LLVM_ABI MachineFunction { return LandingPads; } + /// Forget the landing pads: a target turned them into ordinary blocks and + /// there is no exception table to emit. + void clearLandingPads() { LandingPads.clear(); } + /// Provide the begin and end labels of an invoke style call and associate it /// with a try landing pad block. void addInvoke(MachineBasicBlock *LandingPad, diff --git a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp index a230cc5f0a9272..ec167a680cdc1b 100644 --- a/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp +++ b/llvm/lib/Target/BPF/AsmParser/BPFAsmParser.cpp @@ -235,8 +235,10 @@ struct BPFOperand : public MCParsedAsmOperand { .Case("gotol", true) .Case("gotox", true) .Case("may_goto", true) + .Case("unwind_goto", true) .Case("*", true) .Case("exit", true) + .Case("unwind_exit", true) .Case("lock", true) .Case("ld_pseudo", true) .Case("store_release", true) diff --git a/llvm/lib/Target/BPF/BPF.h b/llvm/lib/Target/BPF/BPF.h index f6103d4d7ad456..fb4a744dad8325 100644 --- a/llvm/lib/Target/BPF/BPF.h +++ b/llvm/lib/Target/BPF/BPF.h @@ -35,6 +35,8 @@ class BPFCheckAndAdjustIRPass }; ModulePass *createBPFCheckAndAdjustIRLegacyPass(); +ModulePass *createBPFUnwindPreparePass(); +FunctionPass *createBPFUnwindLowerPass(); class BPFISelDAGToDAGPass : public SelectionDAGISelPass { public: diff --git a/llvm/lib/Target/BPF/BPFAsmPrinter.cpp b/llvm/lib/Target/BPF/BPFAsmPrinter.cpp index 97b602d866f615..1648474e6b6f8b 100644 --- a/llvm/lib/Target/BPF/BPFAsmPrinter.cpp +++ b/llvm/lib/Target/BPF/BPFAsmPrinter.cpp @@ -205,61 +205,6 @@ void BPFAsmPrinter::emitInstruction(const MachineInstr *MI) { EmitToStreamer(*OutStreamer, TmpInst); } -void BPFAsmPrinter::emitFunctionBodyEnd() { - // Emit .bpf_cleanup section with a flat table of - // (call_site, landing_pad) pairs. - const std::vector<LandingPadInfo> &LandingPads = MF->getLandingPads(); - if (LandingPads.empty()) - return; - - MCContext &Ctx = OutContext; - auto *CleanupSec = - Ctx.getELFSection(".bpf_cleanup", ELF::SHT_PROGBITS, ELF::SHF_ALLOC); - OutStreamer->switchSection(CleanupSec); - - const auto &TypeInfos = MF->getTypeInfos(); - const Function &F = MF->getFunction(); - LLVMContext &LLVMCtx = F.getContext(); - - // Each landing pad has BeginLabels/EndLabels marking the invoke - // call sites that unwind to it. - for (const LandingPadInfo &LP : LandingPads) { - // BPF treats all landing pads as catch-all: the kernel redirects to - // the landing pad regardless of exception type. Reject type-specific - // catches and filters which would silently misbehave. - for (int TId : LP.TypeIds) { - if (TId > 0 && TypeInfos[TId - 1] != nullptr) { - LLVMCtx.diagnose(DiagnosticInfoUnsupported( - F, "BPF does not support type-specific exception catches yet")); - return; - } - if (TId < 0) { - LLVMCtx.diagnose(DiagnosticInfoUnsupported( - F, "BPF does not support exception filters yet")); - return; - } - } - - MCSymbol *LPLabel = LP.LandingPadLabel; - if (!LPLabel) - continue; - for (unsigned i = 0, e = LP.BeginLabels.size(); i != e; ++i) { - MCSymbol *Begin = LP.BeginLabels[i]; - MCSymbol *End = LP.EndLabels[i]; - - // Each entry is 3 x 4 bytes: begin, end, landing_pad. - // The invoke region [begin, end) may include argument setup - // before the call. The runtime checks begin <= PC < end. - OutStreamer->emitSymbolValue(Begin, 4); - OutStreamer->emitSymbolValue(End, 4); - OutStreamer->emitSymbolValue(LPLabel, 4); - } - } - - // Switch back to the function's section. - OutStreamer->switchSection(MF->getSection()); -} - MCSymbol *BPFAsmPrinter::getJTPublicSymbol(unsigned JTI) { SmallString<60> Name; raw_svector_ostream(Name) diff --git a/llvm/lib/Target/BPF/BPFAsmPrinter.h b/llvm/lib/Target/BPF/BPFAsmPrinter.h index 86028ee7fe8a70..2730871734535d 100644 --- a/llvm/lib/Target/BPF/BPFAsmPrinter.h +++ b/llvm/lib/Target/BPF/BPFAsmPrinter.h @@ -35,7 +35,6 @@ class BPFAsmPrinter : public AsmPrinter { const char *ExtraCode, raw_ostream &O) override; void emitInstruction(const MachineInstr *MI) override; - void emitFunctionBodyEnd() override; MCSymbol *getJTPublicSymbol(unsigned JTI); void emitJumpTableInfo() override; diff --git a/llvm/lib/Target/BPF/BPFInstrInfo.td b/llvm/lib/Target/BPF/BPFInstrInfo.td index 6799d44f8ab796..8c3e09009a5e50 100644 --- a/llvm/lib/Target/BPF/BPFInstrInfo.td +++ b/llvm/lib/Target/BPF/BPFInstrInfo.td @@ -243,7 +243,7 @@ class JMP_IND<BPFJumpOp Opc, string OpcodeStr, list<dag> Pattern> let BPFClass = BPF_JMP; } -class JMP_JCOND<BPFJumpOp Opc, string OpcodeStr, list<dag> Pattern> +class JMP_JCOND<BPFJumpOp Opc, bits<4> Kind, string OpcodeStr, list<dag> Pattern> : TYPE_ALU_JMP<Opc.Value, BPF_K.Value, (outs), (ins brtarget:$BrDst), @@ -251,6 +251,7 @@ class JMP_JCOND<BPFJumpOp Opc, string OpcodeStr, list<dag> Pattern> Pattern> { bits<16> BrDst; + let Inst{55-52} = Kind; let Inst{47-32} = BrDst; let BPFClass = BPF_JMP; } @@ -307,7 +308,9 @@ defm JULE : J<BPF_JLE, "<=", BPF_CC_LEU, BPF_CC_LEU_32>; defm JSLT : J<BPF_JSLT, "s<", BPF_CC_LT, BPF_CC_LT_32>; defm JSLE : J<BPF_JSLE, "s<=", BPF_CC_LE, BPF_CC_LE_32>; defm JSET : J<BPF_JSET, "&", NoCond, NoCond>; -def JCOND : JMP_JCOND<BPF_JCOND, "may_goto", []>; +def JCOND : JMP_JCOND<BPF_JCOND, 0, "may_goto", []>; +// Follows a call: taken when the callee left with unwind_exit. +def UNWIND_GOTO : JMP_JCOND<BPF_JCOND, 1, "unwind_goto", []>; let Predicates = [BPFHasGotox] in { let isIndirectBranch = 1, isBarrier = 1 in { @@ -829,19 +832,22 @@ class NOP_I<string OpcodeStr> let hasSideEffects = 0, isCodeGenOnly = 1 in def NOP : NOP_I<"nop">; -class RET<string OpcodeStr> +class RET<bits<4> Kind, string OpcodeStr, list<dag> Pattern> : TYPE_ALU_JMP<BPF_EXIT.Value, BPF_K.Value, (outs), (ins), !strconcat(OpcodeStr, ""), - [(BPFretglue)]> { + Pattern> { + let Inst{55-52} = Kind; let Inst{31-0} = 0; let BPFClass = BPF_JMP; } let isReturn = 1, isTerminator = 1, hasDelaySlot=0, isBarrier = 1, isNotDuplicable = 1 in { - def RET : RET<"exit">; + def RET : RET<0, "exit", [(BPFretglue)]>; + // Like exit, except that unwind_goto after the call is taken. + def UNWIND_EXIT : RET<1, "unwind_exit", []>; } // ADJCALLSTACKDOWN/UP pseudo insns diff --git a/llvm/lib/Target/BPF/BPFMIPeephole.cpp b/llvm/lib/Target/BPF/BPFMIPeephole.cpp index 43682453b9f4ec..bd834f0d4b72d3 100644 --- a/llvm/lib/Target/BPF/BPFMIPeephole.cpp +++ b/llvm/lib/Target/BPF/BPFMIPeephole.cpp @@ -494,11 +494,14 @@ bool BPFMIPreEmitPeepholeImpl::adjustBranch() { } const BasicBlock *TermBB = MBB->getBasicBlock(); + // 'if r1 == r2 goto' has three operands, 'unwind_goto' has one. + MachineOperand &CondTarget = + CondJmp->getOperand(CondJmp->getNumExplicitOperands() - 1); int Dist; // (3). conditional jmp to another bb or follow through. if (!UncondJmp) { - CondTargetBB = CondJmp->getOperand(2).getMBB(); + CondTargetBB = CondTarget.getMBB(); MachineBasicBlock *FollowBB = FollowThroughBB[MBB]; Dist = SoFarNumInsns[CondTargetBB] - CondTargetBB->size() - CurrNumInsns; if (in16BitRange(Dist)) @@ -527,16 +530,7 @@ bool BPFMIPreEmitPeepholeImpl::adjustBranch() { MF->insert(MBB_I, New_B1); // replace B2 cond jump - if (CondJmp->getOperand(1).isReg()) - BuildMI(*MBB, MachineBasicBlock::iterator(*CondJmp), CondJmp->getDebugLoc(), TII->get(CondJmp->getOpcode())) - .addReg(CondJmp->getOperand(0).getReg()) - .addReg(CondJmp->getOperand(1).getReg()) - .addMBB(New_B1); - else - BuildMI(*MBB, MachineBasicBlock::iterator(*CondJmp), CondJmp->getDebugLoc(), TII->get(CondJmp->getOpcode())) - .addReg(CondJmp->getOperand(0).getReg()) - .addImm(CondJmp->getOperand(1).getImm()) - .addMBB(New_B1); + CondTarget.setMBB(New_B1); // it is possible that CondTargetBB and FollowBB are the same. But the // above Dist checking should already filtered this case. @@ -552,13 +546,12 @@ bool BPFMIPreEmitPeepholeImpl::adjustBranch() { New_B0->addSuccessor(FollowBB); New_B1->addSuccessor(CondTargetBB); - CondJmp->eraseFromParent(); Changed = true; continue; } // (4). conditional jmp followed by an unconditional jmp. - CondTargetBB = CondJmp->getOperand(2).getMBB(); + CondTargetBB = CondTarget.getMBB(); JmpBB = UncondJmp->getOperand(0).getMBB(); // We have @@ -587,16 +580,7 @@ bool BPFMIPreEmitPeepholeImpl::adjustBranch() { MF->insert(++MBB->getIterator(), New_B); // replace B2 cond jump - if (CondJmp->getOperand(1).isReg()) - BuildMI(*MBB, MachineBasicBlock::iterator(*CondJmp), CondJmp->getDebugLoc(), TII->get(CondJmp->getOpcode())) - .addReg(CondJmp->getOperand(0).getReg()) - .addReg(CondJmp->getOperand(1).getReg()) - .addMBB(New_B); - else - BuildMI(*MBB, MachineBasicBlock::iterator(*CondJmp), CondJmp->getDebugLoc(), TII->get(CondJmp->getOpcode())) - .addReg(CondJmp->getOperand(0).getReg()) - .addImm(CondJmp->getOperand(1).getImm()) - .addMBB(New_B); + CondTarget.setMBB(New_B); if (CondTargetBB != JmpBB) MBB->removeSuccessor(CondTargetBB); @@ -606,7 +590,6 @@ bool BPFMIPreEmitPeepholeImpl::adjustBranch() { BuildMI(New_B, CondJmp->getDebugLoc(), TII->get(BPF::JMPL)).addMBB(CondTargetBB); New_B->addSuccessor(CondTargetBB); - CondJmp->eraseFromParent(); Changed = true; } diff --git a/llvm/lib/Target/BPF/BPFTargetMachine.cpp b/llvm/lib/Target/BPF/BPFTargetMachine.cpp index d502eb220a92e4..30d166de7451ab 100644 --- a/llvm/lib/Target/BPF/BPFTargetMachine.cpp +++ b/llvm/lib/Target/BPF/BPFTargetMachine.cpp @@ -117,6 +117,7 @@ void BPFPassConfig::addIRPasses() { addPass(createBPFCheckAndAdjustIRLegacyPass()); TargetPassConfig::addIRPasses(); + addPass(createBPFUnwindPreparePass()); } TargetTransformInfo @@ -147,6 +148,7 @@ void BPFPassConfig::addMachineSSAOptimization() { } void BPFPassConfig::addPreEmitPass() { + addPass(createBPFUnwindLowerPass()); addPass(createBPFMIPreEmitCheckingLegacyPass()); if (!DisableMIPeephole) { addPass(createBPFMIExpandStackArgPseudosLegacyPass()); diff --git a/llvm/lib/Target/BPF/BPFUnwind.cpp b/llvm/lib/Target/BPF/BPFUnwind.cpp new file mode 100644 index 00000000000000..5979a93d185d83 --- /dev/null +++ b/llvm/lib/Target/BPF/BPFUnwind.cpp @@ -0,0 +1,297 @@ +//===-- BPFUnwind.cpp - Lower unwinding to unwind_goto and unwind_exit ----===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// There are two ways to leave a frame, and a call that may unwind is followed +// by an insn that tells them apart: +// ret exit +// invoke f() unwind pad call f; unwind_goto pad +// resume r0 = 0; unwind_exit +// call _Unwind_RaiseException r0 = 0; unwind_exit +// invoke _Unwind_RaiseException goto pad +// unwind_goto is taken when the callee left with unwind_exit. No register says +// so: how one insn tells the other is up to the JIT. Otherwise unwind_exit is +// exit, hence r0. A pad expects the exception object in r0. There is none, so +// a pad that looks at it starts with r0 = 0. +// +// _Unwind_RaiseException() is how a language runtime starts an unwind with the +// Itanium ABI, e.g. at the end of a panic handler. To start one and to carry +// one on is the same thing here: leave the frame with unwind_exit, after its +// own pad if it has one. It cannot fail, so it does not return. +// +// There are no tables and nothing for a loader to do. +// +//===----------------------------------------------------------------------===// + +#include "BPF.h" +#include "BPFInstrInfo.h" +#include "BPFSubtarget.h" +#include "llvm/ADT/DepthFirstIterator.h" +#include "llvm/ADT/SetVector.h" +#include "llvm/CodeGen/MachineFunctionPass.h" +#include "llvm/CodeGen/MachineInstrBuilder.h" +#include "llvm/IR/DiagnosticInfo.h" +#include "llvm/IR/IRBuilder.h" +#include "llvm/IR/InstIterator.h" +#include "llvm/IR/IntrinsicInst.h" +#include "llvm/IR/Module.h" +#include "llvm/Transforms/Utils/Local.h" + +using namespace llvm; + +/// On other targets it is a function of the unwinder library, which is why +/// nothing else in LLVM knows the name. +static constexpr StringLiteral UnwindRaise = "_Unwind_RaiseException"; + +/// Not being nounwind is the default in IR, so that alone is not enough: +/// uwtable is how a frontend asks for unwinding through a function to work. +static bool canUnwind(const Function &F) { + return F.hasUWTable() && !F.doesNotThrow(); +} + +namespace { + +/// Make every call that may unwind an invoke: it ends its block and has a pad. +struct BPFUnwindPrepare : public ModulePass { + static char ID; + BPFUnwindPrepare() : ModulePass(ID) {} + StringRef getPassName() const override { return "BPF Unwind Prepare"; } + + static bool isRaise(const CallBase &CB) { + const auto *F = dyn_cast<Function>(CB.getCalledOperand()); + return F && F->isDeclaration() && F->getName() == UnwindRaise; + } + + static bool mayUnwind(const Instruction &I) { + const auto *CB = dyn_cast<CallBase>(&I); + return CB && !CB->doesNotThrow() && !CB->isInlineAsm() && + !isa<IntrinsicInst>(CB); + } + + bool runOnModule(Module &M) override { + bool Changed = false; + + for (Function &F : M) { + if (F.isDeclaration() || !canUnwind(F)) + continue; + + LLVMContext &C = M.getContext(); + SmallVector<CallInst *, 8> Calls; + for (Instruction &I : instructions(F)) { + auto *CB = dyn_cast<CallBase>(&I); + if (CB && isRaise(*CB)) { + // Nobody looks at the exception object. + for (Use &Arg : CB->args()) + Arg.set(PoisonValue::get(Arg->getType())); + CB->setDoesNotReturn(); + // Without a pad it leaves the frame. + if (isa<CallInst>(CB)) + CB->setDoesNotThrow(); + Changed = true; + } else if (isa<CallInst>(I) && mayUnwind(I)) { + Calls.push_back(cast<CallInst>(&I)); + } else if (auto *RI = dyn_cast<ResumeInst>(&I)) { + // DwarfEHPrepare passes it to _Unwind_Resume(). Don't keep it alive + // over the pad for that. + RI->setOperand(0, PoisonValue::get(RI->getValue()->getType())); + Changed = true; + } + } + if (Calls.empty()) + continue; + + if (!F.hasPersonalityFn()) + F.setPersonalityFn(cast<Constant>( + M.getOrInsertFunction("__gcc_personality_v0", + FunctionType::get(Type::getInt32Ty(C), true)) + .getCallee())); + BasicBlock *Pad = BasicBlock::Create(C, "unwind.resume", &F); + IRBuilder<> B(Pad); + LandingPadInst *LP = B.CreateLandingPad( + StructType::get(PointerType::getUnqual(C), Type::getInt32Ty(C)), 0); + LP->setCleanup(true); + B.CreateResume(PoisonValue::get(LP->getType())); + for (CallInst *CI : Calls) + changeToInvokeAndSplitBasicBlock(CI, Pad); + Changed = true; + } + return Changed; + } +}; + +/// After register allocation, which took call -> pad for an EH edge and so +/// keeps only r6-r9 and the stack alive over it, turn the edge into a branch. +struct BPFUnwindLower : public MachineFunctionPass { + static char ID; + BPFUnwindLower() : MachineFunctionPass(ID) {} + StringRef getPassName() const override { return "BPF Unwind Lower"; } + + /// What DwarfEHPrepare made of 'resume'. + StringRef UnwindResume; + + bool leavesFrame(const MachineInstr &MI) const { + const MachineOperand &MO = MI.getOperand(0); + // A program that defines them gets what it defines. + if (!MO.isGlobal() || !MO.getGlobal()->isDeclaration()) + return false; + StringRef Name = MO.getGlobal()->getName(); + return Name == UnwindRaise || Name == UnwindResume; + } + + /// Nothing after \p MI in its block runs: \p Succ, if any, is what follows. + static void endBlockAfter(MachineInstr &MI, MachineBasicBlock *Succ) { + MachineBasicBlock &MBB = *MI.getParent(); + MBB.erase(std::next(MI.getIterator()), MBB.end()); + while (!MBB.succ_empty()) + MBB.removeSuccessor(MBB.succ_begin()); + if (Succ) + MBB.addSuccessor(Succ); + } + + static const char *unsupported(const MachineFunction &MF) { + if (MF.getLandingPads().empty()) + return nullptr; + // Not canUnwind(): a function that catches everything is nounwind. + if (!MF.getFunction().hasUWTable()) + return "BPF needs uwtable on a function with a landing pad"; + // Every pad is a catch-all: the branch is taken whatever was thrown. + for (const LandingPadInfo &LP : MF.getLandingPads()) { + for (int TId : LP.TypeIds) { + if (TId > 0 && MF.getTypeInfos()[TId - 1]) + return "BPF does not support type-specific exception catches yet"; + // An empty filter lets nothing through, so it is a catch-all too. rustc + // guards every function that must not unwind with one. + if (TId < 0 && MF.getFilterIds()[-1 - TId]) + return "BPF does not support exception filters yet"; + } + } + return nullptr; + } + + bool runOnMachineFunction(MachineFunction &MF) override { + const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo(); + const TargetRegisterInfo &TRI = *MF.getSubtarget().getRegisterInfo(); + UnwindResume = MF.getSubtarget().getTargetLowering()->getLibcallName( + RTLIB::UNWIND_RESUME); + DenseMap<MCSymbol *, MachineBasicBlock *> Begin; + SmallVector<std::pair<MachineInstr *, MachineBasicBlock *>, 8> Sites; + SmallVector<MachineInstr *, 8> Labels, Resumes; + SmallSetVector<MachineBasicBlock *, 8> Pads; + + if (const char *Msg = unsupported(MF)) { + const Function &F = MF.getFunction(); + F.getContext().diagnose(DiagnosticInfoUnsupported(F, Msg)); + return false; + } + + for (const LandingPadInfo &LP : MF.getLandingPads()) + for (MCSymbol *Sym : LP.BeginLabels) + Begin[Sym] = LP.LandingPadBlock; + + for (MachineBasicBlock &MBB : MF) { + MachineBasicBlock *Pad = nullptr; + for (MachineInstr &MI : MBB) { + if (MI.isEHLabel()) { + Pad = Begin.lookup(MI.getOperand(0).getMCSymbol()); + Labels.push_back(&MI); + } else if (MI.isCall() && Pad) { + Sites.push_back({&MI, Pad}); + } else if (MI.isCall() && leavesFrame(MI)) { + Resumes.push_back(&MI); + } + } + } + if (Labels.empty() && Resumes.empty()) + return false; + // There is no unwind_goto after the calls of it. + if (!Resumes.empty() && !canUnwind(MF.getFunction())) { + const Function &F = MF.getFunction(); + F.getContext().diagnose(DiagnosticInfoUnsupported( + F, "BPF cannot unwind out of a function that is nounwind or has no " + "uwtable")); + return false; + } + for (MachineInstr *MI : Labels) + MI->eraseFromParent(); + + for (auto [Call, Pad] : Sites) { + MachineBasicBlock *MBB = Call->getParent(); + DebugLoc DL = Call->getDebugLoc(); + const auto *II = + dyn_cast_or_null<InvokeInst>(MBB->getBasicBlock()->getTerminator()); + + Pads.insert(Pad); + if (leavesFrame(*Call)) { + endBlockAfter(*Call, Pad); + BuildMI(MBB, DL, TII.get(BPF::JMP)).addMBB(Pad); + Call->eraseFromParent(); + } else if (II && II->doesNotReturn()) { + // Nothing sensible follows the call, but something has to. + endBlockAfter(*Call, Pad); + BuildMI(MBB, DL, TII.get(BPF::UNWIND_GOTO)).addMBB(Pad); + BuildMI(MBB, DL, TII.get(BPF::JMP)).addMBB(Pad); + } else { + MachineBasicBlock *Rest = MBB->splitAt(*Call); + if (Rest != MBB) { + Rest->removeSuccessor(Pad); + MBB->addSuccessor(Pad); + } + BuildMI(MBB, DL, TII.get(BPF::UNWIND_GOTO)).addMBB(Pad); + } + } + + for (MachineBasicBlock *Pad : Pads) { + if (Pad->computeRegisterLiveness(&TRI, BPF::R0, Pad->begin()) != + MachineBasicBlock::LQR_Dead) + BuildMI(*Pad, Pad->begin(), DebugLoc(), TII.get(BPF::MOV_ri), BPF::R0) + .addImm(0); + Pad->removeLiveIn(BPF::R0); + } + + for (MachineInstr *Call : Resumes) { + MachineBasicBlock *MBB = Call->getParent(); + DebugLoc DL = Call->getDebugLoc(); + + BuildMI(*MBB, Call, DL, TII.get(BPF::MOV_ri), BPF::R0).addImm(0); + MachineInstr *Exit = BuildMI(*MBB, Call, DL, TII.get(BPF::UNWIND_EXIT)) + .addReg(BPF::R0, RegState::Implicit); + endBlockAfter(*Exit, nullptr); + } + + // The pads are plain blocks now and there is no table to emit. + for (MachineBasicBlock &MBB : MF) + MBB.setIsEHPad(false); + MF.clearLandingPads(); + + // What was behind a call that does not return. It can be a loop. + df_iterator_default_set<MachineBasicBlock *> Reachable; + for (MachineBasicBlock *MBB : depth_first_ext(&MF, Reachable)) + (void)MBB; + for (MachineBasicBlock &MBB : make_early_inc_range(MF)) { + if (Reachable.count(&MBB) || MBB.hasAddressTaken()) + continue; + while (!MBB.succ_empty()) + MBB.removeSuccessor(MBB.succ_begin()); + MBB.eraseFromParent(); + } + for (MachineBasicBlock &MBB : MF) + if (!MBB.empty() && MBB.back().getOpcode() == BPF::JMP && + MBB.isLayoutSuccessor(MBB.back().getOperand(0).getMBB())) + MBB.back().eraseFromParent(); + return true; + } +}; + +} // namespace + +char BPFUnwindPrepare::ID = 0; +char BPFUnwindLower::ID = 0; + +ModulePass *llvm::createBPFUnwindPreparePass() { + return new BPFUnwindPrepare(); +} +FunctionPass *llvm::createBPFUnwindLowerPass() { return new BPFUnwindLower(); } diff --git a/llvm/lib/Target/BPF/CMakeLists.txt b/llvm/lib/Target/BPF/CMakeLists.txt index ddff969890eff9..94e0a059abc1bd 100644 --- a/llvm/lib/Target/BPF/CMakeLists.txt +++ b/llvm/lib/Target/BPF/CMakeLists.txt @@ -41,6 +41,7 @@ add_llvm_target(BPFCodeGen BPFSubtarget.cpp BPFTargetLoweringObjectFile.cpp BPFTargetMachine.cpp + BPFUnwind.cpp BPFMIPeephole.cpp BPFMIChecking.cpp BPFMISimplifyPatchable.cpp diff --git a/llvm/lib/Target/BPF/MCTargetDesc/BPFMCTargetDesc.cpp b/llvm/lib/Target/BPF/MCTargetDesc/BPFMCTargetDesc.cpp index 5f44dd9583aff3..75297e3ae6544f 100644 --- a/llvm/lib/Target/BPF/MCTargetDesc/BPFMCTargetDesc.cpp +++ b/llvm/lib/Target/BPF/MCTargetDesc/BPFMCTargetDesc.cpp @@ -82,7 +82,8 @@ class BPFMCInstrAnalysis : public MCInstrAnalysis { // The target is the 3rd operand of cond inst and the 1st of uncond inst. int32_t Imm; if (isConditionalBranch(Inst)) { - if (Inst.getOpcode() == BPF::JCOND) + if (Inst.getOpcode() == BPF::JCOND || + Inst.getOpcode() == BPF::UNWIND_GOTO) Imm = (short)Inst.getOperand(0).getImm(); else Imm = (short)Inst.getOperand(2).getImm(); diff --git a/llvm/test/CodeGen/BPF/cleanup-section.ll b/llvm/test/CodeGen/BPF/cleanup-section.ll deleted file mode 100644 index bf215461ddb3d5..00000000000000 --- a/llvm/test/CodeGen/BPF/cleanup-section.ll +++ /dev/null @@ -1,160 +0,0 @@ -; RUN: llc -mtriple=bpfel -mcpu=v4 -filetype=asm -o - %s | FileCheck --check-prefix=ASM %s -; RUN: llc -mtriple=bpfel -mcpu=v4 -filetype=obj -o %t %s -; RUN: llvm-readelf -S %t | FileCheck --check-prefix=SECTIONS %s -; RUN: llvm-readelf -r %t | FileCheck --check-prefix=RELOCS %s - -; Verify that invoke/landingpad with cleanup produces a .bpf_cleanup section -; containing (call_site, landing_pad) pairs with R_BPF_64_NODYLD32 relocations. -; Models a Rust BTreeMap insertion that may fail, with Drop cleanup -; calling bpf_free to release allocated nodes. - -declare ptr @bpf_alloc(i64, i64) -declare void @bpf_free(ptr) -declare void @btree_insert(ptr, ptr, i64) -declare i32 @rust_personality(i32, i64, ptr, ptr) - -; SECTIONS: .bpf_cleanup - -define void @stat_inc(ptr %map, ptr %key, i64 %val) personality ptr @rust_personality { -entry: - %node = call ptr @bpf_alloc(i64 256, i64 0) - %null = icmp eq ptr %node, null - br i1 %null, label %oom, label %insert - -insert: - invoke void @btree_insert(ptr %map, ptr %node, i64 %val) - to label %done unwind label %cleanup - -done: - ret void - -cleanup: - %lp = landingpad { ptr, i32 } cleanup - call void @bpf_free(ptr %node) - resume { ptr, i32 } %lp - -oom: - ret void -} - -; ASM-LABEL: stat_inc: -; ASM: r1 = 256 -; ASM: call bpf_alloc -; ASM: if r0 == 0 goto -; Normal path: call btree_insert then exit -; ASM: .Ltmp0: -; ASM: call btree_insert -; ASM: .Ltmp1: -; ASM: exit -; Cleanup landing pad: free the node, then resume unwinding -; ASM: .Ltmp2: -; ASM: r1 = r7 -; ASM: call bpf_free -; ASM: call _Unwind_Resume -; .bpf_cleanup entry: [.Ltmp0, .Ltmp1) -> .Ltmp2 -; ASM: .section .bpf_cleanup,"a",@progbits -; ASM-NEXT: .long .Ltmp0 -; ASM-NEXT: .long .Ltmp1 -; ASM-NEXT: .long .Ltmp2 - -define void @two_allocs(ptr %map) personality ptr @rust_personality { -entry: - %n1 = call ptr @bpf_alloc(i64 256, i64 0) - invoke void @btree_insert(ptr %map, ptr %n1, i64 1) - to label %second unwind label %clean1 - -second: - %n2 = call ptr @bpf_alloc(i64 512, i64 0) - invoke void @btree_insert(ptr %map, ptr %n2, i64 2) - to label %done unwind label %clean2 - -done: - ret void - -clean2: - ; Must free both n2 and n1 - %lp2 = landingpad { ptr, i32 } cleanup - call void @bpf_free(ptr %n2) - call void @bpf_free(ptr %n1) - resume { ptr, i32 } %lp2 - -clean1: - ; Only n1 allocated at this point - %lp1 = landingpad { ptr, i32 } cleanup - call void @bpf_free(ptr %n1) - resume { ptr, i32 } %lp1 -} - -; ASM-LABEL: two_allocs: -; First alloc + invoke -; ASM: r1 = 256 -; ASM: call bpf_alloc -; ASM: .Ltmp3: -; ASM: call btree_insert -; ASM: .Ltmp4: -; Second alloc + invoke -; ASM: r1 = 512 -; ASM: call bpf_alloc -; ASM: .Ltmp6: -; ASM: call btree_insert -; ASM: .Ltmp7: -; ASM: exit -; clean2: free n2, then fall through to free n1 -; ASM: .Ltmp8: -; ASM: call bpf_free -; clean1 entry point, shared tail: free n1 then resume -; ASM: .Ltmp5: -; ASM: call bpf_free -; ASM: call _Unwind_Resume -; .bpf_cleanup entries: two triples -; ASM: .section .bpf_cleanup,"a",@progbits -; ASM-NEXT: .long .Ltmp3 -; ASM-NEXT: .long .Ltmp4 -; ASM-NEXT: .long .Ltmp5 -; ASM-NEXT: .long .Ltmp6 -; ASM-NEXT: .long .Ltmp7 -; ASM-NEXT: .long .Ltmp8 - -; Models std::panic::catch_unwind: catch the exception, free resources, -; and return normally. The landing pad uses "catch ptr null" (catch-all) -; instead of "cleanup", and does NOT call _Unwind_Resume — unwinding stops here. - -define i64 @with_catch_unwind(ptr %map, i64 %val) personality ptr @rust_personality { -entry: - %node = call ptr @bpf_alloc(i64 128, i64 0) - invoke void @btree_insert(ptr %map, ptr %node, i64 %val) - to label %ok unwind label %caught - -ok: - ret i64 0 - -caught: - %lp = landingpad { ptr, i32 } - catch ptr null - call void @bpf_free(ptr %node) - ret i64 1 -} - -; ASM-LABEL: with_catch_unwind: -; ASM: r1 = 128 -; ASM: call bpf_alloc -; ASM: .Ltmp9: -; ASM: call btree_insert -; ASM: .Ltmp10: -; ASM: r0 = 0 -; ASM: exit -; Catch block: free the node, return 1 (no _Unwind_Resume) -; ASM: .Ltmp11: -; ASM: call bpf_free -; ASM: r0 = 1 -; ASM: exit -; .bpf_cleanup entry for the catch -; ASM: .section .bpf_cleanup,"a",@progbits -; ASM-NEXT: .long .Ltmp9 -; ASM-NEXT: .long .Ltmp10 -; ASM-NEXT: .long .Ltmp11 - -; .bpf_cleanup should have R_BPF_64_NODYLD32 relocations -; 4 entries x 3 fields = 12 relocs -; RELOCS: .rel.bpf_cleanup -; RELOCS-COUNT-12: R_BPF_64_NODYLD32 {{.*}} .text diff --git a/llvm/test/CodeGen/BPF/unwind-far-pad.ll b/llvm/test/CodeGen/BPF/unwind-far-pad.ll new file mode 100644 index 00000000000000..7d1684f7ba9954 --- /dev/null +++ b/llvm/test/CodeGen/BPF/unwind-far-pad.ll @@ -0,0 +1,36 @@ +; RUN: llc -mtriple=bpfel -mcpu=v4 -gotol-abs-low-bound 0 -filetype=asm -o - %s | FileCheck %s + +; unwind_goto has a 16-bit offset like other conditional jumps, and it has to stay +; right after the call. + +declare void @may_unwind() +declare void @bpf_free(ptr) nounwind +declare i32 @rust_eh_personality(i32, i64, ptr, ptr) + +define i64 @far(ptr %node) uwtable personality ptr @rust_eh_personality { +entry: + invoke void @may_unwind() + to label %done unwind label %cleanup + +done: + ret i64 7 + +cleanup: + %lp = landingpad { ptr, i32 } cleanup + call void @bpf_free(ptr %node) + resume { ptr, i32 } %lp +} + +; CHECK-LABEL: far: +; CHECK: call may_unwind +; CHECK-NEXT: unwind_goto [[HOP:.LBB[0-9_]+]] +; CHECK: goto [[DONE:.LBB[0-9_]+]] +; CHECK: [[HOP]]: +; CHECK-NEXT: gotol [[PAD:.LBB[0-9_]+]] +; CHECK: [[DONE]]: +; CHECK-NEXT: r0 = 7 +; CHECK-NEXT: exit +; CHECK: [[PAD]]: +; CHECK: call bpf_free +; CHECK-NEXT: r0 = 0 +; CHECK-NEXT: unwind_exit diff --git a/llvm/test/CodeGen/BPF/unwind-own-raise.ll b/llvm/test/CodeGen/BPF/unwind-own-raise.ll new file mode 100644 index 00000000000000..54b9eff29035f7 --- /dev/null +++ b/llvm/test/CodeGen/BPF/unwind-own-raise.ll @@ -0,0 +1,20 @@ +; RUN: llc -mtriple=bpfel -mcpu=v4 -filetype=asm -o - %s | FileCheck %s + +; A program that defines _Unwind_RaiseException() gets what it defines: the call +; is an ordinary one. + +define i32 @_Unwind_RaiseException(ptr %exn) nounwind uwtable { +entry: + ret i32 5 +} + +define i32 @caller() nounwind uwtable { +entry: + %r = call i32 @_Unwind_RaiseException(ptr null) + ret i32 %r +} + +; CHECK-LABEL: caller: +; CHECK: call _Unwind_RaiseException +; CHECK-NOT: unwind_exit +; CHECK: exit diff --git a/llvm/test/CodeGen/BPF/unwind-reject-filter.ll b/llvm/test/CodeGen/BPF/unwind-reject-filter.ll new file mode 100644 index 00000000000000..9f5879dbb3eef7 --- /dev/null +++ b/llvm/test/CodeGen/BPF/unwind-reject-filter.ll @@ -0,0 +1,24 @@ +; RUN: not llc -mtriple=bpfel -mcpu=v4 -filetype=asm -o - %s 2>&1 | FileCheck %s + +; A filter that lets some types through needs type information at run time. +; The empty one is a catch-all and is accepted, see unwind.ll. + +@_ZTIi = external constant ptr + +declare void @may_throw() +declare i32 @rust_eh_personality(i32, i64, ptr, ptr) + +define void @typed_filter() uwtable personality ptr @rust_eh_personality { +entry: + invoke void @may_throw() + to label %ok unwind label %lpad + +ok: + ret void + +lpad: + %lp = landingpad { ptr, i32 } filter [1 x ptr] [ptr @_ZTIi] + ret void +} + +; CHECK: error: {{.*}} BPF does not support exception filters yet diff --git a/llvm/test/CodeGen/BPF/unwind-reject-no-uwtable.ll b/llvm/test/CodeGen/BPF/unwind-reject-no-uwtable.ll new file mode 100644 index 00000000000000..cee925997f7b23 --- /dev/null +++ b/llvm/test/CodeGen/BPF/unwind-reject-no-uwtable.ll @@ -0,0 +1,22 @@ +; RUN: not llc -mtriple=bpfel -mcpu=v4 -filetype=asm -o - %s 2>&1 | FileCheck %s + +; Only functions with uwtable take part in unwinding. A landing pad in +; a function without it would never be reached. + +declare void @may_throw() +declare i32 @rust_eh_personality(i32, i64, ptr, ptr) + +define void @no_uwtable() personality ptr @rust_eh_personality { +entry: + invoke void @may_throw() + to label %ok unwind label %lpad + +ok: + ret void + +lpad: + %lp = landingpad { ptr, i32 } cleanup + resume { ptr, i32 } %lp +} + +; CHECK: error: {{.*}} BPF needs uwtable on a function with a landing pad diff --git a/llvm/test/CodeGen/BPF/unwind-reject-nounwind.ll b/llvm/test/CodeGen/BPF/unwind-reject-nounwind.ll new file mode 100644 index 00000000000000..12238bad1997df --- /dev/null +++ b/llvm/test/CodeGen/BPF/unwind-reject-nounwind.ll @@ -0,0 +1,14 @@ +; RUN: not llc -mtriple=bpfel -mcpu=v4 -filetype=asm -o - %s 2>&1 | FileCheck %s + +; There is no unwind_goto after a call of a nounwind function, so an unwind +; cannot start there. + +declare i32 @_Unwind_RaiseException(ptr) + +define void @cannot() nounwind uwtable { +entry: + %err = call i32 @_Unwind_RaiseException(ptr null) + unreachable +} + +; CHECK: error: {{.*}} BPF cannot unwind out of a function that is nounwind or has no uwtable diff --git a/llvm/test/CodeGen/BPF/cleanup-reject-typed-catch.ll b/llvm/test/CodeGen/BPF/unwind-reject-typed-catch.ll similarity index 80% rename from llvm/test/CodeGen/BPF/cleanup-reject-typed-catch.ll rename to llvm/test/CodeGen/BPF/unwind-reject-typed-catch.ll index 81973a0ae3a48c..06dd66942ea019 100644 --- a/llvm/test/CodeGen/BPF/cleanup-reject-typed-catch.ll +++ b/llvm/test/CodeGen/BPF/unwind-reject-typed-catch.ll @@ -6,9 +6,9 @@ @_ZTIi = external constant ptr declare void @may_throw() -declare i32 @rust_personality(i32, i64, ptr, ptr) +declare i32 @rust_eh_personality(i32, i64, ptr, ptr) -define void @typed_catch() personality ptr @rust_personality { +define void @typed_catch() uwtable personality ptr @rust_eh_personality { entry: invoke void @may_throw() to label %ok unwind label %lpad diff --git a/llvm/test/CodeGen/BPF/unwind.ll b/llvm/test/CodeGen/BPF/unwind.ll new file mode 100644 index 00000000000000..435d3ed318e14c --- /dev/null +++ b/llvm/test/CodeGen/BPF/unwind.ll @@ -0,0 +1,298 @@ +; RUN: llc -mtriple=bpfel -mcpu=v4 -filetype=asm -o - %s | FileCheck %s +; RUN: llc -mtriple=bpfel -mcpu=v4 -filetype=obj -o %t %s +; RUN: llvm-readelf -S -s %t | FileCheck --check-prefix=OBJ %s + +; A function leaves with exit or with unwind_exit, and unwind_goto after the call +; tells them apart. There are no tables and no calls into an unwinder. +; A frontend asks for that with uwtable, like rustc does with -C panic=unwind. + +; OBJ-NOT: .bpf_cleanup +; OBJ-NOT: .gcc_except_table +; OBJ-NOT: _Unwind_ +; OBJ-NOT: abort +; OBJ-NOT: personality + +declare ptr @bpf_alloc(i64, i64) nounwind +declare void @bpf_free(ptr) nounwind +declare void @btree_insert(ptr, ptr, i64) +declare i32 @_Unwind_RaiseException(ptr) +declare void @_Unwind_Resume(ptr) noreturn +declare void @abort() noreturn nounwind +declare i32 @rust_eh_personality(i32, i64, ptr, ptr) + +; invoke, cleanup pad, resume. A call of nounwind function is left alone. +define void @stat_inc(ptr %map, i64 %val) uwtable personality ptr @rust_eh_personality { +entry: + %node = call ptr @bpf_alloc(i64 256, i64 0) + invoke void @btree_insert(ptr %map, ptr %node, i64 %val) + to label %done unwind label %cleanup + +done: + ret void + +cleanup: + %lp = landingpad { ptr, i32 } cleanup + call void @bpf_free(ptr %node) + resume { ptr, i32 } %lp +} + +; CHECK-LABEL: stat_inc: +; CHECK: call bpf_alloc +; CHECK-NOT: unwind_goto +; CHECK: call btree_insert +; CHECK-NEXT: unwind_goto [[PAD:.LBB[0-9_]+]] +; CHECK-NEXT: # %bb +; CHECK-NEXT: exit +; CHECK: [[PAD]]: +; CHECK: call bpf_free +; CHECK-NEXT: r0 = 0 +; CHECK-NEXT: unwind_exit + +; A plain call that may unwind: nothing to clean up, pass it on. +define i64 @passes_on(ptr %map) uwtable { +entry: + call void @btree_insert(ptr %map, ptr null, i64 1) + ret i64 7 +} + +; CHECK-LABEL: passes_on: +; CHECK: call btree_insert +; CHECK-NEXT: unwind_goto [[PAD:.LBB[0-9_]+]] +; CHECK-NEXT: # %bb +; CHECK-NEXT: r0 = 7 +; CHECK-NEXT: exit +; CHECK: [[PAD]]: +; CHECK-NEXT: r0 = 0 +; CHECK-NEXT: unwind_exit + +; catch_unwind: the pad does not resume. +define i64 @catches(ptr %map) uwtable personality ptr @rust_eh_personality { +entry: + invoke void @btree_insert(ptr %map, ptr null, i64 1) + to label %ok unwind label %caught + +ok: + ret i64 0 + +caught: + %lp = landingpad { ptr, i32 } catch ptr null + ret i64 1 +} + +; CHECK-LABEL: catches: +; CHECK: call btree_insert +; CHECK-NEXT: unwind_goto [[PAD:.LBB[0-9_]+]] +; CHECK-NEXT: # %bb +; CHECK-NEXT: r0 = 0 +; CHECK-NEXT: exit +; CHECK: [[PAD]]: +; CHECK-NEXT: r0 = 1 +; CHECK-NEXT: exit + +; A pad expects the exception object in r0. There is none. +define void @looks_at_exn(ptr %map) uwtable personality ptr @rust_eh_personality { +entry: + invoke void @btree_insert(ptr %map, ptr null, i64 1) + to label %ok unwind label %caught + +ok: + ret void + +caught: + %lp = landingpad { ptr, i32 } catch ptr null + %exn = extractvalue { ptr, i32 } %lp, 0 + call void @bpf_free(ptr %exn) + ret void +} + +; CHECK-LABEL: looks_at_exn: +; CHECK: call btree_insert +; CHECK-NEXT: unwind_goto [[PAD:.LBB[0-9_]+]] +; CHECK: [[PAD]]: +; CHECK-NEXT: r0 = 0 +; CHECK-NEXT: r1 = r0 +; CHECK-NEXT: call bpf_free + +; The same after inference: it is nounwind, so there is no unwind_goto after the +; calls of it, but it still needs one after the call of its callee. +define i64 @catches_nounwind(ptr %map) nounwind uwtable personality ptr @rust_eh_personality { +entry: + invoke void @btree_insert(ptr %map, ptr null, i64 1) + to label %ok unwind label %caught + +ok: + ret i64 0 + +caught: + %lp = landingpad { ptr, i32 } catch ptr null + ret i64 1 +} + +; CHECK-LABEL: catches_nounwind: +; CHECK: call btree_insert +; CHECK-NEXT: unwind_goto +; CHECK-NOT: unwind_exit +; CHECK: .Lfunc_end + +; An empty filter lets nothing through: one more way to say catch-all. That is +; how rustc guards a function that must not unwind, e.g. an extern "C" one. +define i64 @empty_filter(ptr %map) nounwind uwtable personality ptr @rust_eh_personality { +entry: + invoke void @btree_insert(ptr %map, ptr null, i64 1) + to label %ok unwind label %terminate + +ok: + ret i64 0 + +terminate: + %lp = landingpad { ptr, i32 } filter [0 x ptr] zeroinitializer + ret i64 2 +} + +; CHECK-LABEL: empty_filter: +; CHECK: call btree_insert +; CHECK-NEXT: unwind_goto [[PAD:.LBB[0-9_]+]] +; CHECK: [[PAD]]: +; CHECK-NEXT: r0 = 2 + +; The start of an unwind, as at the end of a panic handler. It is not different +; from carrying one on. By its prototype _Unwind_RaiseException() comes back +; when it fails, so a runtime has something after it. Here it cannot fail: that +; code is dropped and the function ends with unwind_exit. +define void @fail() noreturn uwtable { +entry: + %err = call i32 @_Unwind_RaiseException(ptr null) + call void @abort() + unreachable +} + +; CHECK-LABEL: fail: +; CHECK-NOT: call +; CHECK: r0 = 0 +; CHECK-NEXT: unwind_exit +; CHECK-NOT: call +; CHECK: .Lfunc_end + +; The same in a frame that owns something: its own pad comes first. +define void @fail_owner(ptr %node) uwtable personality ptr @rust_eh_personality { +entry: + %err = invoke i32 @_Unwind_RaiseException(ptr null) + to label %failed unwind label %cleanup + +failed: + call void @abort() + unreachable + +cleanup: + %lp = landingpad { ptr, i32 } cleanup + call void @bpf_free(ptr %node) + resume { ptr, i32 } %lp +} + +; CHECK-LABEL: fail_owner: +; CHECK-NOT: _Unwind_ +; CHECK-NOT: exit +; CHECK: call bpf_free +; CHECK-NEXT: r0 = 0 +; CHECK-NEXT: unwind_exit +; CHECK-NOT: call +; CHECK: .Lfunc_end + +; What is behind it can be a loop, e.g. 'loop {}' in Rust. The verifier rejects +; unreachable code, so it has to go as well. +define void @fail_then_spins() noreturn uwtable { +entry: + %err = call i32 @_Unwind_RaiseException(ptr null) + br label %spin + +spin: + br label %spin +} + +; CHECK-LABEL: fail_then_spins: +; CHECK: r0 = 0 +; CHECK-NEXT: unwind_exit +; CHECK-NOT: goto +; CHECK: .Lfunc_end + +; 'resume' spelled as the call that other targets lower it to. +define void @resume_spelled_out(ptr %exn) uwtable { +entry: + call void @_Unwind_Resume(ptr %exn) + unreachable +} + +; CHECK-LABEL: resume_spelled_out: +; CHECK-NOT: call +; CHECK: r0 = 0 +; CHECK-NEXT: unwind_exit + +; A callee that does not return: its unwind_exit needs unwind_goto all the same. +; What follows is not reached, but the block cannot end there. +define i32 @check(i32 %i) uwtable { +entry: + %c = icmp ugt i32 %i, 7 + br i1 %c, label %bad, label %good + +bad: + call void @fail() + unreachable + +good: + ret i32 %i +} + +; CHECK-LABEL: check: +; CHECK: call fail +; CHECK-NEXT: unwind_goto [[PAD:.LBB[0-9_]+]] +; CHECK-NEXT: goto [[PAD]] +; CHECK: [[PAD]]: +; CHECK-NEXT: r0 = 0 +; CHECK-NEXT: unwind_exit + +; An indirect call is followed by unwind_goto whatever the callee is. A nounwind +; callee does not have to know. +define internal i32 @plain(i32 %x) nounwind uwtable { + %r = mul i32 %x, 3 + ret i32 %r +} + +@ops = constant [1 x ptr] [ptr @plain] + +define i32 @indirect(i64 %i, i32 %x) uwtable { +entry: + %p = getelementptr [1 x ptr], ptr @ops, i64 0, i64 %i + %f = load volatile ptr, ptr %p + %r = call i32 %f(i32 %x) + ret i32 %r +} + +; CHECK-LABEL: plain: +; CHECK: w0 = w1 +; CHECK-NEXT: w0 *= 3 +; CHECK-NEXT: exit +; CHECK-LABEL: indirect: +; CHECK: callx +; CHECK-NEXT: unwind_goto + +; nounwind: nothing changes. +define i32 @leaf(i32 %x) nounwind uwtable { + %p = call ptr @bpf_alloc(i64 8, i64 0) + call void @bpf_free(ptr %p) + ret i32 %x +} + +; CHECK-LABEL: leaf: +; CHECK-NOT: unwind_ +; CHECK: exit + +; Not being nounwind is the default in IR. Without uwtable nothing changes. +define i32 @not_asked(ptr %map) { + call void @btree_insert(ptr %map, ptr null, i64 1) + ret i32 0 +} + +; CHECK-LABEL: not_asked: +; CHECK: call btree_insert +; CHECK-NEXT: {{[rw]}}0 = 0 +; CHECK-NEXT: exit diff --git a/llvm/test/MC/BPF/insn-unwind.s b/llvm/test/MC/BPF/insn-unwind.s new file mode 100644 index 00000000000000..73e41afcbb7312 --- /dev/null +++ b/llvm/test/MC/BPF/insn-unwind.s @@ -0,0 +1,23 @@ +# RUN: llvm-mc -triple bpfel -filetype=obj -o %t %s +# RUN: llvm-objdump --no-print-imm-hex -d -r %t | FileCheck %s +# RUN: llvm-mc -triple bpfeb -filetype=obj -o %t %s +# RUN: llvm-objdump --no-print-imm-hex -d -r %t | FileCheck --check-prefix=CHECK-BE %s + +// src_reg tells unwind_goto from may_goto and unwind_exit from exit. + + call 1 + unwind_goto Lpad // BPF_JCOND | BPF_K, src_reg = 1 + may_goto Lpad // BPF_JCOND | BPF_K, src_reg = 0 + exit // BPF_EXIT +Lpad: + unwind_exit // BPF_EXIT, src_reg = 1 + +// CHECK: e5 10 02 00 00 00 00 00 unwind_goto +2 +// CHECK: e5 00 01 00 00 00 00 00 may_goto +1 +// CHECK: 95 00 00 00 00 00 00 00 exit +// CHECK: 95 10 00 00 00 00 00 00 unwind_exit + +// CHECK-BE: e5 01 00 02 00 00 00 00 unwind_goto +2 +// CHECK-BE: e5 00 00 01 00 00 00 00 may_goto +1 +// CHECK-BE: 95 00 00 00 00 00 00 00 exit +// CHECK-BE: 95 01 00 00 00 00 00 00 unwind_exit _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
