[
https://issues.apache.org/jira/browse/SOLR-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128264#comment-18128264
]
Nick Shanin commented on SOLR-18551:
------------------------------------
Following up on the replication question: I ran a standalone benchmark of three
Directory variants on a 3M-document index (warm page cache, Lucene 10.4, JDK
21, 2,000 term queries per run, 3 runs each).
* Copy time: about the same for all three (4.5–5.5 s). The copy path writes, so
this is expected.
* Open time: NIOFS is faster (420–530 ms, against 800–900 ms for MMap).
* Query latency: NIOFS is about 13x slower. Average 875–1000 µs against 65–78
µs for MMap, and p99 3.7–6.6 ms against about 325 µs.
These are warm-cache numbers only. I recommend that we do not switch from
MMapDirectory to NIOFSDirectory for replication.
I created SOLR-18554 for the IOContext audit.
> IndexFetcher can leak "mmaps" on TLOG/PULL non-leaders
> ------------------------------------------------------
>
> Key: SOLR-18551
> URL: https://issues.apache.org/jira/browse/SOLR-18551
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10.1
> Reporter: Jason Gerlowski
> Priority: Major
>
> MMapDirectory works by mapping index files into a process's virtual address
> space, but deleting these maps later on can an expensive operation for the
> JVM. Lucene, starting in 9.12, has dealt with this expense by lumping
> mappings together in "arenas" which can then be unmapped in bulk later. This
> changes the mmap lifecycle a bit: individual mmaps are added to the arena on
> creation and now persist until the entire arena is closed at some point later
> (typically when the DirectoryReader itself is closed).
> When DirectoryReaders and SegmentReaders (and the arena's they own) are
> long-lived, these mmaps can accumulate and eventually exhaust the OS mmap
> count or even the virtual-memory for a Solr process. This doesn't happen
> normally because IndexWriter, IndexReader, etc. are conscientious in the
> files that they open, and because SegmentReaders don't tend to be open all
> that long.
> But this has caused leaks in Lucene
> ([example|https://github.com/apache/lucene/issues/15068]) and I've found a
> leak scenario in Solr 9.10.1 now too.
> *Solr Leak Scenario*
> The particular path I've seen that triggers this leak in Solr is around the
> periodic polling done by TLOG and PULL replicas.
> Each time a TLOG or PULL replica polls the leader and there are new changes,
> the leader will provide a list of its index files and the follower will call
> "[IndexFetcher.slowFileExists|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1337-L1344]";
> to see which of the files are already present locally. "slowFileExists"
> calls {{Directory.openInput(fileName, IOContext.DEFAULT)}} as a quick way to
> see if the file is present. But when MMapDirectory is in use, this tiny
> existence check creates a (possibly large) mmap that may stick around for
> quite awhile - often until the segment is merged away and the
> SegmentReader/arena are closed.
> This existence-check leak:
> # happens for each leader index file that the follower has to check for...
> # and repeats every polling cycle.
> So the leak adds up pretty quickly, especially with decent sized indices.
> I've had some customers hit Solr process's mmap limit because of this leak.
> I've also seen it exhaust the process's virtual memory space (*128TB!*) with
> a comparatively small index of ~100gb.
> The issue tends to be worse on large indices that receive a trickle of
> updates: both because they have more index files and because their larger
> index files are infrequently involved in merges so they have more time for
> the leak to accumulate.
> *Solution*
> The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
> use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
> altogether. Given that slowFileExists doesn't care about the file content at
> all it seems wasteful to even create a mmap for it in the first place and
> there's probably a more efficient way to check its existence.
> Lucene's arena-pooling also gets much better in 10.x, as they tweak some
> defaults related to this leak behavior. We should consider adopting those
> changes in 9.x by setting
> {{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
> Lucene default in 10.x due to all the problems this pooling behavior raised
> in their 9.12.x line). See
> [here|https://github.com/apache/lucene/issues/15068] for more context on that
> sysprop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
