spyffe created this revision.
spyffe added reviewers: a.sidorin, beanz, loladiro, v.g.vassilev.
spyffe added a subscriber: cfe-commits.
spyffe set the repository for this revision to rL LLVM.
Herald added a subscriber: mgorny.
LLVM's JIT is now the foundation of dynamic-compilation features for many
languages. Clang also has low-level support for dynamic compilation
(`ASTImporter` and `ExternalASTSource`, notably). How the compiler is set up
for dynamic parsing is generally left up to individual clients, for example
LLDB's C/C++/Objective-C expression parser and the ROOT project.
Although this arrangement offers external clients the flexibility to implement
dynamic features as they see fit, the lack of an in-tree client means that
subtle bugs can be introduced that cause regressions in the external clients
but aren't caught by tests (or users) until much later. LLDB for example
regularly encounters complicated ODR violation scenarios where it is not
immediately clear who is at fault.
I propose a simple expression parser be added to Clang. I aim to have it
encompass two main features:
- It should be able to look up external declarations from a variety of sources
(e.g., from previous dynamic compilations, from modules, or from DWARF) and
have clear conflict resolution rules with easily understood errors. This
functionality will be supported by in-tree tests.
- It should work hand in hand with the LLVM JIT to resolve the locations of
external declarations so that e.g. variables can be redeclared and (for
high-performance applications like DTrace) external variables can be accessed
directly from the registers where they reside.
I have attached a tester that parses a sequence of source files and then uses
them as source data for an expression. External references are resolved using
an `ExternalASTSource` that responds to name queries using an `ASTImporter`.
This is the setup that LLDB uses, and the motivating reason for `MinimalImport`
in `ASTImporter`.
Over time my intention is to make this more complete and support the many
scenarios that LLDB can support. I also want to identify places where LLDB
does the wrong thing and we can make this functionality more correct, hopefully
eliminating frustrating and opaque errors. I also want to spot where Clang
might get things wrong, and be able to point Clang developers to an
easy-to-reproduce, in-tree test that doesn't require external projects or
patches. Finally, I want to identify how we can make this functionality as
generic as possible for the current clients besides LLDB, and for potential
future clients.
Repository:
rL LLVM
https://reviews.llvm.org/D27180
Files:
test/Import/empty-struct/Inputs/S.c
test/Import/empty-struct/test.c
tools/CMakeLists.txt
tools/clang-import-test/CMakeLists.txt
tools/clang-import-test/clang-import-test.cpp
Index: tools/clang-import-test/clang-import-test.cpp
===================================================================
--- tools/clang-import-test/clang-import-test.cpp
+++ tools/clang-import-test/clang-import-test.cpp
@@ -0,0 +1,347 @@
+//===-- import-test.cpp - ASTImporter/ExternalASTSource testbed -----------===//
+//
+// The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/AST/ASTContext.h"
+#include "clang/AST/ASTImporter.h"
+#include "clang/Basic/Builtins.h"
+#include "clang/Basic/IdentifierTable.h"
+#include "clang/Basic/SourceLocation.h"
+#include "clang/Basic/TargetInfo.h"
+#include "clang/Basic/TargetOptions.h"
+#include "clang/CodeGen/ModuleBuilder.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/TextDiagnosticBuffer.h"
+#include "clang/Lex/Lexer.h"
+#include "clang/Lex/Preprocessor.h"
+#include "clang/Parse/ParseAST.h"
+
+#include "llvm/IR/LLVMContext.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Host.h"
+#include "llvm/Support/Signals.h"
+
+#include <memory>
+#include <string>
+
+using namespace clang;
+
+static llvm::cl::opt<std::string> Expression(
+ "expression", llvm::cl::Required,
+ llvm::cl::desc("Path to a file containing the expression to parse"));
+
+static llvm::cl::list<std::string>
+ Imports("import", llvm::cl::ZeroOrMore,
+ llvm::cl::desc("Path to a file containing declarations to import"));
+
+static llvm::cl::list<std::string>
+ ClangArgs("-Xcc", llvm::cl::ZeroOrMore,
+ llvm::cl::desc("Argument to pass to the CompilerInvocation"),
+ llvm::cl::CommaSeparated);
+
+static llvm::cl::opt<bool> LogLookups(
+ "log-lookups",
+ llvm::cl::desc("Print each lookup performed on behalf of the expression"));
+
+namespace {
+
+class TestDiagnosticConsumer : public DiagnosticConsumer {
+private:
+ std::unique_ptr<TextDiagnosticBuffer> Passthrough;
+ const LangOptions *LangOpts = nullptr;
+
+public:
+ TestDiagnosticConsumer()
+ : Passthrough(llvm::make_unique<TextDiagnosticBuffer>()) {}
+
+ virtual void BeginSourceFile(const LangOptions &LangOpts,
+ const Preprocessor *PP = nullptr) override {
+ this->LangOpts = &LangOpts;
+ return Passthrough->BeginSourceFile(LangOpts, PP);
+ }
+
+ virtual void EndSourceFile() override {
+ this->LangOpts = nullptr;
+ Passthrough->EndSourceFile();
+ }
+
+ virtual void finish() override { Passthrough->finish(); }
+
+ virtual bool IncludeInDiagnosticCounts() const override {
+ return Passthrough->IncludeInDiagnosticCounts();
+ }
+
+private:
+ static void PrintSourceForLocation(const SourceLocation &Loc,
+ SourceManager &SM) {
+ bool Invalid = true;
+ const char *LocData = SM.getCharacterData(Loc, &Invalid);
+ if (Invalid) {
+ return;
+ }
+ unsigned LocColumn = SM.getSpellingColumnNumber(Loc, &Invalid) - 1;
+ if (Invalid) {
+ return;
+ }
+ FileID FID = SM.getFileID(Loc);
+ llvm::MemoryBuffer *Buffer = SM.getBuffer(FID, Loc, &Invalid);
+ if (Invalid) {
+ return;
+ }
+
+ assert(LocData >= Buffer->getBufferStart() &&
+ LocData < Buffer->getBufferEnd());
+
+ const char *LineBegin = LocData - LocColumn;
+
+ if (LineBegin < Buffer->getBufferStart()) {
+ LineBegin = Buffer->getBufferStart();
+ }
+
+ const char *LineEnd = nullptr;
+
+ for (LineEnd = LineBegin; *LineEnd != '\n' && *LineEnd != '\r' &&
+ LineEnd < Buffer->getBufferEnd();
+ ++LineEnd)
+ ;
+
+ std::string LineString(LineBegin, LineEnd - LineBegin);
+
+ fprintf(stderr, "%s\n", LineString.c_str());
+ std::string Space(LocColumn, ' ');
+ fprintf(stderr, "%s^\n", Space.c_str());
+ }
+
+ virtual void HandleDiagnostic(DiagnosticsEngine::Level DiagLevel,
+ const Diagnostic &Info) override {
+ if (Info.hasSourceManager() && LangOpts) {
+ SourceManager &SM = Info.getSourceManager();
+
+ if (Info.getLocation().isValid()) {
+ Info.getLocation().print(llvm::errs(), SM);
+ llvm::errs() << ": ";
+ }
+
+ SmallVector<char, 16> DiagText;
+ Info.FormatDiagnostic(DiagText);
+ DiagText.push_back('\0');
+ llvm::errs() << DiagText.data() << '\n';
+
+ if (Info.getLocation().isValid()) {
+ PrintSourceForLocation(Info.getLocation(), SM);
+ }
+
+ for (const CharSourceRange &Range : Info.getRanges()) {
+ bool Invalid = true;
+ StringRef Ref = Lexer::getSourceText(Range, SM, *LangOpts, &Invalid);
+ if (!Invalid) {
+ llvm::errs() << Ref.str().c_str() << '\n';
+ }
+ }
+ }
+ DiagnosticConsumer::HandleDiagnostic(DiagLevel, Info);
+ }
+};
+
+class TestExternalASTSource : public ExternalASTSource {
+private: llvm::ArrayRef<CompilerInstance *> ImportCIs;
+ std::map<CompilerInstance *, std::unique_ptr<ASTImporter>> ForwardImporters;
+ std::map<CompilerInstance *, std::unique_ptr<ASTImporter>> ReverseImporters;
+
+public:
+ TestExternalASTSource(CompilerInstance &ExpressionCI,
+ llvm::ArrayRef<CompilerInstance *> ImportCIs)
+ : ImportCIs(ImportCIs) {
+ for (CompilerInstance *ImportCI : ImportCIs) {
+ const bool MinimalImport = true;
+ ForwardImporters.emplace(std::make_pair(
+ ImportCI,
+ llvm::make_unique<ASTImporter>(
+ ExpressionCI.getASTContext(), ExpressionCI.getFileManager(),
+ ImportCI->getASTContext(), ImportCI->getFileManager(),
+ MinimalImport)));
+ ReverseImporters.emplace(std::make_pair(
+ ImportCI,
+ llvm::make_unique<ASTImporter>(
+ ImportCI->getASTContext(), ImportCI->getFileManager(),
+ ExpressionCI.getASTContext(), ExpressionCI.getFileManager(),
+ MinimalImport)));
+ }
+ }
+
+ bool FindExternalVisibleDeclsByName(const DeclContext *DC,
+ DeclarationName Name) override {
+ std::vector<NamedDecl *> Decls;
+
+ if (llvm::isa<TranslationUnitDecl>(DC)) {
+ for (CompilerInstance *I : ImportCIs) {
+ DeclarationName FromName = ReverseImporters[I]->Import(Name);
+ DeclContextLookupResult Result =
+ I->getASTContext().getTranslationUnitDecl()->lookup(FromName);
+ for (NamedDecl *FromD : Result) {
+ NamedDecl *D = llvm::cast<NamedDecl>(ForwardImporters[I]->Import(FromD));
+ Decls.push_back(D);
+ }
+ }
+ }
+ if (LogLookups) {
+ if (auto ND = llvm::dyn_cast<NamedDecl>(DC)) {
+ llvm::outs() << "[log-lookups] (in " << DC->getDeclKindName() << " "
+ << ND->getName() << ") " << Name;
+ } else {
+ llvm::outs() << "[log-lookups] (in a " << DC->getDeclKindName() << ") "
+ << Name;
+ }
+ if (Decls.empty()) {
+ llvm::outs() << " -> {}" << '\n';
+ } else {
+ llvm::outs() << +" -> {" << '\n';
+ for (NamedDecl *Decl : Decls) {
+ llvm::outs() << +"[log-lookups] . " << *Decl << '\n';
+ }
+ llvm::outs() << +"[log-lookups] }" << '\n';
+ }
+ }
+ if (Decls.empty()) {
+ return false;
+ } else {
+ SetExternalVisibleDeclsForName(DC, Name, Decls);
+ return true;
+ }
+ }
+
+ void
+ FindExternalLexicalDecls(const DeclContext *DC,
+ llvm::function_ref<bool(Decl::Kind)> IsKindWeWant,
+ SmallVectorImpl<Decl *> &Result) override {
+
+ }
+};
+
+std::unique_ptr<CompilerInstance> BuildCompilerInstance() {
+ auto Ins = llvm::make_unique<CompilerInstance>();
+ auto DC = llvm::make_unique<TestDiagnosticConsumer>();
+ const bool ShouldOwnClient = true;
+ Ins->createDiagnostics(DC.release(), ShouldOwnClient);
+
+ auto Inv = llvm::make_unique<CompilerInvocation>();
+
+ std::vector<const char *> ClangArgv(ClangArgs.size());
+ std::transform(ClangArgs.begin(), ClangArgs.end(), ClangArgv.begin(),
+ [](std::string &s) -> const char * { return s.data(); });
+
+ CompilerInvocation::CreateFromArgs(*Inv, ClangArgv.data(),
+ &ClangArgv.data()[ClangArgv.size()],
+ Ins->getDiagnostics());
+
+ Inv->getLangOpts()->CPlusPlus = true;
+ Inv->getLangOpts()->CPlusPlus11 = true;
+ Inv->getHeaderSearchOpts().UseLibcxx = true;
+ Inv->getLangOpts()->Bool = true;
+ Inv->getLangOpts()->WChar = true;
+ Inv->getLangOpts()->Blocks = true;
+ Inv->getLangOpts()->DebuggerSupport = true;
+ Inv->getLangOpts()->SpellChecking = false;
+ Inv->getLangOpts()->ThreadsafeStatics = false;
+ Inv->getLangOpts()->AccessControl = false;
+ Inv->getLangOpts()->DollarIdents = true;
+ Inv->getCodeGenOpts().setDebugInfo(codegenoptions::FullDebugInfo);
+ Inv->getTargetOpts().Triple = llvm::sys::getDefaultTargetTriple();
+
+ Ins->setInvocation(Inv.release());
+
+ TargetInfo *TI = TargetInfo::CreateTargetInfo(
+ Ins->getDiagnostics(), Ins->getInvocation().TargetOpts);
+ Ins->setTarget(TI);
+ Ins->getTarget().adjust(Ins->getLangOpts());
+ Ins->createFileManager();
+ Ins->createSourceManager(Ins->getFileManager());
+ Ins->createPreprocessor(TU_Complete);
+
+ return Ins;
+}
+
+std::unique_ptr<ASTContext>
+BuildASTContext(CompilerInstance &CI, SelectorTable &ST, Builtin::Context &BC) {
+ auto AST = llvm::make_unique<ASTContext>(
+ CI.getLangOpts(), CI.getSourceManager(),
+ CI.getPreprocessor().getIdentifierTable(), ST, BC);
+ AST->InitBuiltinTypes(CI.getTarget());
+ return AST;
+}
+
+void AddExternalSource(CompilerInstance &CI, llvm::ArrayRef<CompilerInstance *> Imports) {
+ ASTContext &AST = CI.getASTContext();
+ auto ES = llvm::make_unique<TestExternalASTSource>(CI, Imports);
+ AST.setExternalSource(ES.release());
+ AST.getTranslationUnitDecl()->setHasExternalVisibleStorage();
+}
+
+std::unique_ptr<CodeGenerator> BuildCodeGen(CompilerInstance &CI,
+ llvm::LLVMContext &LLVMCtx) {
+ std::string ModuleName("$__module");
+ return std::unique_ptr<CodeGenerator>(CreateLLVMCodeGen(
+ CI.getDiagnostics(), ModuleName, CI.getHeaderSearchOpts(),
+ CI.getPreprocessorOpts(), CI.getCodeGenOpts(), LLVMCtx));
+}
+
+bool ParseSource(const std::string &Path, CompilerInstance &CI,
+ CodeGenerator &CG) {
+ SourceManager &SM = CI.getSourceManager();
+ const FileEntry *FE = CI.getFileManager().getFile(Path);
+ if (!FE) {
+ llvm::errs() << "Couldn't open " << Path << '\n';
+ return false;
+ }
+ SM.setMainFileID(SM.createFileID(FE, SourceLocation(), SrcMgr::C_User));
+ ParseAST(CI.getPreprocessor(), &CG, CI.getASTContext());
+ return true;
+}
+
+bool Parse(const std::string &Path, std::unique_ptr<CompilerInstance> &CI,
+ llvm::ArrayRef<CompilerInstance *> Imports) {
+ CI = BuildCompilerInstance();
+ auto ST = llvm::make_unique<SelectorTable>();
+ auto BC = llvm::make_unique<Builtin::Context>();
+ std::unique_ptr<ASTContext> AST = BuildASTContext(*CI, *ST, *BC);
+ CI->setASTContext(AST.release());
+ AddExternalSource(*CI, Imports);
+
+ auto LLVMCtx = llvm::make_unique<llvm::LLVMContext>();
+ std::unique_ptr<CodeGenerator> CG = BuildCodeGen(*CI, *LLVMCtx);
+ CG->Initialize(CI->getASTContext());
+
+ CI->getDiagnosticClient().BeginSourceFile(CI->getLangOpts(),
+ &CI->getPreprocessor());
+ if (!ParseSource(Path, *CI, *CG)) {
+ return false;
+ }
+ CI->getDiagnosticClient().EndSourceFile();
+ return (CI->getDiagnosticClient().getNumErrors() == 0);
+}
+}
+
+int main(int argc, const char **argv) {
+ const bool DisableCrashReporting = true;
+ llvm::sys::PrintStackTraceOnErrorSignal(argv[0], DisableCrashReporting);
+ llvm::cl::ParseCommandLineOptions(argc, argv);
+ std::vector<std::unique_ptr<CompilerInstance>> ImportCIs(Imports.size());
+ for (auto I : llvm::zip(Imports, ImportCIs)) {
+ if (!Parse(std::get<0>(I), std::get<1>(I), {})) {
+ exit(-1);
+ }
+ }
+ std::vector<CompilerInstance *> UnownedCIs(Imports.size());
+ std::transform(ImportCIs.begin(), ImportCIs.end(), UnownedCIs.begin(),
+ std::mem_fn(&std::unique_ptr<CompilerInstance>::get));
+ std::unique_ptr<CompilerInstance> ExpressionCI;
+ if (!Parse(Expression, ExpressionCI, UnownedCIs)) {
+ exit(-1);
+ }
+ return 0;
+}
Index: tools/clang-import-test/CMakeLists.txt
===================================================================
--- tools/clang-import-test/CMakeLists.txt
+++ tools/clang-import-test/CMakeLists.txt
@@ -0,0 +1,18 @@
+set(LLVM_LINK_COMPONENTS
+ support
+)
+
+add_clang_tool(clang-import-test
+ clang-import-test.cpp
+ )
+
+set(CLANG_IMPORT_TEST_LIB_DEPS
+ clangAST
+ clangBasic
+ clangCodeGen
+ clangFrontend
+ )
+
+target_link_libraries(clang-import-test
+ ${CLANG_IMPORT_TEST_LIB_DEPS}
+ )
\ No newline at end of file
Index: tools/CMakeLists.txt
===================================================================
--- tools/CMakeLists.txt
+++ tools/CMakeLists.txt
@@ -5,6 +5,7 @@
add_clang_subdirectory(clang-format)
add_clang_subdirectory(clang-format-vs)
add_clang_subdirectory(clang-fuzzer)
+add_clang_subdirectory(clang-import-test)
add_clang_subdirectory(clang-offload-bundler)
add_clang_subdirectory(c-index-test)
Index: test/Import/empty-struct/test.c
===================================================================
--- test/Import/empty-struct/test.c
+++ test/Import/empty-struct/test.c
@@ -0,0 +1,5 @@
+// RUN: clang-import-test -import %S/Inputs/S.c -expression %s
+void expr() {
+ struct S MyS;
+ void *MyPtr = &MyS;
+}
Index: test/Import/empty-struct/Inputs/S.c
===================================================================
--- test/Import/empty-struct/Inputs/S.c
+++ test/Import/empty-struct/Inputs/S.c
@@ -0,0 +1,2 @@
+struct S {
+};
_______________________________________________
cfe-commits mailing list
[email protected]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
