[
https://issues.apache.org/jira/browse/SOLR-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128711#comment-18128711
]
Jason Gerlowski commented on SOLR-18551:
----------------------------------------
bq. Curious; how did you find this solution?
At my work we switched a customer over to using TLOG/PULL replicas and within a
few hours certain Solr nodes were seeing lots of "mmap failed" error messages.
Houston happened to know about https://github.com/apache/lucene/issues/15068,
and that provided most of the detail in how "arenas" are used to manage these
mmaps, how that can lead to leaks in some cases, and how IOContext.READONCE
avoids the issue and should be used where appropriate.
So that narrowed things down a good bit. From there it was largely a matter of
figuring out where we use IOContext.DEFAULT that might be TLOG/PULL specific
and then looking through "/proc/<pid>/maps" and a heap dump to validate where
the maps were being consumed/leaked.
> IndexFetcher can leak "mmaps" on TLOG/PULL non-leaders
> ------------------------------------------------------
>
> Key: SOLR-18551
> URL: https://issues.apache.org/jira/browse/SOLR-18551
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10.1
> Reporter: Jason Gerlowski
> Priority: Major
>
> MMapDirectory works by mapping index files into a process's virtual address
> space, but deleting these maps later on can an expensive operation for the
> JVM. Lucene, starting in 9.12, has dealt with this expense by lumping
> mappings together in "arenas" which can then be unmapped in bulk later. This
> changes the mmap lifecycle a bit: individual mmaps are added to the arena on
> creation and now persist until the entire arena is closed at some point later
> (typically when the DirectoryReader itself is closed).
> When DirectoryReaders and SegmentReaders (and the arena's they own) are
> long-lived, these mmaps can accumulate and eventually exhaust the OS mmap
> count or even the virtual-memory for a Solr process. This doesn't happen
> normally because IndexWriter, IndexReader, etc. are conscientious in the
> files that they open, and because SegmentReaders don't tend to be open all
> that long.
> But this has caused leaks in Lucene
> ([example|https://github.com/apache/lucene/issues/15068]) and I've found a
> leak scenario in Solr 9.10.1 now too.
> *Solr Leak Scenario*
> The particular path I've seen that triggers this leak in Solr is around the
> periodic polling done by TLOG and PULL replicas.
> Each time a TLOG or PULL replica polls the leader and there are new changes,
> the leader will provide a list of its index files and the follower will call
> "[IndexFetcher.slowFileExists|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1337-L1344]";
> to see which of the files are already present locally. "slowFileExists"
> calls {{Directory.openInput(fileName, IOContext.DEFAULT)}} as a quick way to
> see if the file is present. But when MMapDirectory is in use, this tiny
> existence check creates a (possibly large) mmap that may stick around for
> quite awhile - often until the segment is merged away and the
> SegmentReader/arena are closed.
> This existence-check leak:
> # happens for each leader index file that the follower has to check for...
> # and repeats every polling cycle.
> So the leak adds up pretty quickly, especially with decent sized indices.
> I've had some customers hit Solr process's mmap limit because of this leak.
> I've also seen it exhaust the process's virtual memory space (*128TB!*) with
> a comparatively small index of ~100gb.
> The issue tends to be worse on large indices that receive a trickle of
> updates: both because they have more index files and because their larger
> index files are infrequently involved in merges so they have more time for
> the leak to accumulate.
> *Solution*
> The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
> use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
> altogether. Given that slowFileExists doesn't care about the file content at
> all it seems wasteful to even create a mmap for it in the first place and
> there's probably a more efficient way to check its existence.
> Lucene's arena-pooling also gets much better in 10.x, as they tweak some
> defaults related to this leak behavior. We should consider adopting those
> changes in 9.x by setting
> {{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
> Lucene default in 10.x due to all the problems this pooling behavior raised
> in their 9.12.x line). See
> [here|https://github.com/apache/lucene/issues/15068] for more context on that
> sysprop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
