matanper opened a new issue, #69860: URL: https://github.com/apache/doris/issues/69860
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4. master is the same: `be/src/storage/index/index_fs_directory.cpp:916` creates each RAM directory file with `_CLNEW lucene::store::RAMFile()`, whose 1 KB buffers come from plain `new[]`; CLucene's `DocumentsWriter` byte/char/posting pools and the BKD writer's points are also plain `new`. ### Description While a segment's inverted index is built (load flush, compaction) and while index compaction merges source indexes (`IndexWriter::indexCompaction` → `mergeTerms` writing `.frq`/`.prx`), the index lives in a `DorisRAMFSDirectory` and in CLucene's indexing buffers. None of these allocations go through Doris' Allocator, so no memory tracker sees them: - the RAM directory keeps every file until the index file writer copies it into the segment's index file, which for a compaction happens only when the compaction finishes, so a large base compaction can hold many GB; - CLucene's in-memory postings grow up to `inverted_index_ram_buffer_size` per column writer, times the number of indexed columns being written concurrently. Because the memory is invisible to `mem_limit` back-pressure, load memory limits and memory GC, a BE can be OOM-killed while its trackers report plenty of headroom. ### Solution 1. Account the RAM directory's buffers on the memory tracker of the task that opened the directory, plus a process-wide bvar, and bound the total with a config (e.g. `inverted_index_ram_dir_max_memory`, default a fraction of `mem_limit`). Past the bound, move a file that is still being written to the directory's on-disk path and write later files there (closed files stay in memory until deleted, since readers may hold them). 2. Track CLucene's indexing buffers on the writing task's tracker, e.g. by measuring the net bytes a thread allocates during each call into the library (jemalloc's per-thread `allocated`/`deallocated` counters) and consuming them on the task's tracker until they are freed. ### How to Reproduce? Run a base compaction over a large tablet with several inverted-indexed string columns (or many concurrent loads into such a table) and compare process RSS with the sum of the memory trackers: the difference grows with the index size and is released only when the compaction or flush finishes. ### Anything Else? The index content does not depend on whether the files are in RAM or on disk. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
