qchateau created this revision.
qchateau added a reviewer: sammccall.
Herald added subscribers: usaxena95, kadircet, arphaman.
qchateau requested review of this revision.
Herald added subscribers: cfe-commits, MaskRay, ilya-biryukov.
Herald added a project: clang.
This diff addresses the issue of the ever increasing memory usage of clangd.
The key to understand what happens is to use `malloc_stats()`: malloc arenas
keep getting bigger, although the actual memory used does not. It seems some
operations while bulding the indices (both dynamic and background) create this
problem. Specifically, 'FileSymbols::update' and 'FileSymbols::buildIndex' seem
especially affected.
This diff adds a call to `malloc_trim(0)` at the end of
`FileSymbols::buildIndex`.
For reference:
https://github.com/clangd/clangd/issues/251
https://github.com/clangd/clangd/issues/115
----------------------------
I'm not sure how much I'm allowed to use GNU extensions but this diff has been
a game changer for me and my 8GB RAM laptop.
In any case, I think I've properly diagnosed to issue. Once trimmed, the actual
memory usage of the process is approximately what clangd reports with the
"memoryUsage" feature.
The solution is either what I suggest here (use `malloc_trim`) or rework a ton
of code, change containers and allocators and hope the problem won't happen
again in the future as we add features.
Repository:
rG LLVM Github Monorepo
https://reviews.llvm.org/D93452
Files:
clang-tools-extra/clangd/index/FileIndex.cpp
Index: clang-tools-extra/clangd/index/FileIndex.cpp
===================================================================
--- clang-tools-extra/clangd/index/FileIndex.cpp
+++ clang-tools-extra/clangd/index/FileIndex.cpp
@@ -32,6 +32,7 @@
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/Optional.h"
#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/ScopeExit.h"
#include "llvm/ADT/StringMap.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/Error.h"
@@ -40,11 +41,17 @@
#include <tuple>
#include <utility>
#include <vector>
+#include <malloc.h>
namespace clang {
namespace clangd {
namespace {
+void trimMemory() {
+ if (malloc_trim(0))
+ vlog("Trimmed memory");
+}
+
SlabTuple indexSymbols(ASTContext &AST, std::shared_ptr<Preprocessor> PP,
llvm::ArrayRef<Decl *> DeclsToIndex,
const MainFileMacros *MacroRefsToIndex,
@@ -263,6 +270,10 @@
std::unique_ptr<SymbolIndex>
FileSymbols::buildIndex(IndexType Type, DuplicateHandling DuplicateHandle,
size_t *Version) {
+ // Building the index often leaves a lof of freed but unreleased memory
+ // (at the OS level): manually trim the memory after the index is built
+ auto _ = llvm::make_scope_exit(trimMemory);
+
std::vector<std::shared_ptr<SymbolSlab>> SymbolSlabs;
std::vector<std::shared_ptr<RefSlab>> RefSlabs;
std::vector<std::shared_ptr<RelationSlab>> RelationSlabs;
Index: clang-tools-extra/clangd/index/FileIndex.cpp
===================================================================
--- clang-tools-extra/clangd/index/FileIndex.cpp
+++ clang-tools-extra/clangd/index/FileIndex.cpp
@@ -32,6 +32,7 @@
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/Optional.h"
#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/ScopeExit.h"
#include "llvm/ADT/StringMap.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/Error.h"
@@ -40,11 +41,17 @@
#include <tuple>
#include <utility>
#include <vector>
+#include <malloc.h>
namespace clang {
namespace clangd {
namespace {
+void trimMemory() {
+ if (malloc_trim(0))
+ vlog("Trimmed memory");
+}
+
SlabTuple indexSymbols(ASTContext &AST, std::shared_ptr<Preprocessor> PP,
llvm::ArrayRef<Decl *> DeclsToIndex,
const MainFileMacros *MacroRefsToIndex,
@@ -263,6 +270,10 @@
std::unique_ptr<SymbolIndex>
FileSymbols::buildIndex(IndexType Type, DuplicateHandling DuplicateHandle,
size_t *Version) {
+ // Building the index often leaves a lof of freed but unreleased memory
+ // (at the OS level): manually trim the memory after the index is built
+ auto _ = llvm::make_scope_exit(trimMemory);
+
std::vector<std::shared_ptr<SymbolSlab>> SymbolSlabs;
std::vector<std::shared_ptr<RefSlab>> RefSlabs;
std::vector<std::shared_ptr<RelationSlab>> RelationSlabs;
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
