[
https://issues.apache.org/jira/browse/SOLR-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128250#comment-18128250
]
Nick Shanin edited comment on SOLR-18551 at 10/17/26 11:36 PM:
---------------------------------------------------------------
Great find! There's a typo in your PR title: it says SOLR-18851, but the ticket
is SOLR-18551 (the commit message has the same typo).
I wrote regression tests for this and measured the leak.
*Leak on 10.x and 9.x.* With a {{DirectoryReader}} open, 50 \{{openInput}}
calls with {{IOContext.DEFAULT}} on a \{{_N.si}} file leave 204,800 bytes
mapped on Lucene 10.4.0 and 9.12.3. The {{fileLength}} path maps nothing, and
the mapping is released when the reader closes. {{IOContext.READONCE}} also
maps nothing, as the ticket suggests. {{sharedArenaMaxPermits=1}} removes the
retention too, but {{64}} does not for 50 opens on either version. I measured
this with a standalone program, not the Solr tests:
[https://gist.github.com/nick-boss-tech/bbdb73c27f36e060dd31af0cacb951ef]
I also ran a test for the above case but did not include it because Solr's test
security policy needs a grant to read \{{/proc/self/maps}}. It would need a
policy change to be included.
*Tests.* The branch
[https://github.com/nick-boss-tech/solr/tree/solr-18551-harden-tests] carries
PR #5173's changes plus a focused {{{}IndexFetcherTest{}}}. With the fix, the
tests pass 3 of 3 on main, branch_10x, and 9.x (on JDK 21, and on JDK 11 for
9.x). On each pre-fix base, with only the test file added, they fail 3 of 3,
each at {{{}openInput{}}}. Steps for adding the test to PR #5173 are in
[https://gist.github.com/nick-boss-tech/8248dc275bfa426faa2b9e61596bd543].
Backport branches for maintainers are described in the same gist.
AI-generated, human-reviewed text.
was (Author: JIRAUSER314749):
Great find! There's a typo in your PR title: it says SOLR-18851, but the ticket
is SOLR-18551 (the commit message has the same typo).
I wrote regression tests for this and measured the leak.
*Leak on 10.x and 9.x.* With a {{DirectoryReader}} open, 50 {{openInput}} calls
with {{IOContext.DEFAULT}} on a {{_N.si}} file leave 204,800 bytes mapped on
Lucene 10.4.0 and 9.12.3. The {{fileLength}} path maps nothing, and the mapping
is released when the reader closes. {{IOContext.READONCE}} also maps nothing,
as the ticket suggests. {{sharedArenaMaxPermits=1}} removes the retention too,
but {{64}} does not for 50 opens on either version. I measured this with a
standalone program, not the Solr tests:
[https://gist.github.com/nick-boss-tech/bbdb73c27f36e060dd31af0cacb951ef]
*Tests.* The branch
[https://github.com/nick-boss-tech/solr/tree/solr-18551-harden-tests] carries
PR #5173's changes plus a focused {{{}IndexFetcherTest{}}}. With the fix, the
tests pass 3 of 3 on main, branch_10x, and 9.x (on JDK 21, and on JDK 11 for
9.x). On each pre-fix base, with only the test file added, they fail 3 of 3,
each at {{{}openInput{}}}. Steps for adding the test to PR #5173 are in
[https://gist.github.com/nick-boss-tech/8248dc275bfa426faa2b9e61596bd543].
Backport branches for maintainers are described in the same gist.
AI-generated, human-reviewed text.
> IndexFetcher can leak "mmaps" on TLOG/PULL non-leaders
> ------------------------------------------------------
>
> Key: SOLR-18551
> URL: https://issues.apache.org/jira/browse/SOLR-18551
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10.1
> Reporter: Jason Gerlowski
> Priority: Major
>
> MMapDirectory works by mapping index files into a process's virtual address
> space, but deleting these maps later on can an expensive operation for the
> JVM. Lucene, starting in 9.12, has dealt with this expense by lumping
> mappings together in "arenas" which can then be unmapped in bulk later. This
> changes the mmap lifecycle a bit: individual mmaps are added to the arena on
> creation and now persist until the entire arena is closed at some point later
> (typically when the DirectoryReader itself is closed).
> When DirectoryReaders and SegmentReaders (and the arena's they own) are
> long-lived, these mmaps can accumulate and eventually exhaust the OS mmap
> count or even the virtual-memory for a Solr process. This doesn't happen
> normally because IndexWriter, IndexReader, etc. are conscientious in the
> files that they open, and because SegmentReaders don't tend to be open all
> that long.
> But this has caused leaks in Lucene
> ([example|https://github.com/apache/lucene/issues/15068]) and I've found a
> leak scenario in Solr 9.10.1 now too.
> *Solr Leak Scenario*
> The particular path I've seen that triggers this leak in Solr is around the
> periodic polling done by TLOG and PULL replicas.
> Each time a TLOG or PULL replica polls the leader and there are new changes,
> the leader will provide a list of its index files and the follower will call
> "[IndexFetcher.slowFileExists|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1337-L1344]";
> to see which of the files are already present locally. "slowFileExists"
> calls {{Directory.openInput(fileName, IOContext.DEFAULT)}} as a quick way to
> see if the file is present. But when MMapDirectory is in use, this tiny
> existence check creates a (possibly large) mmap that may stick around for
> quite awhile - often until the segment is merged away and the
> SegmentReader/arena are closed.
> This existence-check leak:
> # happens for each leader index file that the follower has to check for...
> # and repeats every polling cycle.
> So the leak adds up pretty quickly, especially with decent sized indices.
> I've had some customers hit Solr process's mmap limit because of this leak.
> I've also seen it exhaust the process's virtual memory space (*128TB!*) with
> a comparatively small index of ~100gb.
> The issue tends to be worse on large indices that receive a trickle of
> updates: both because they have more index files and because their larger
> index files are infrequently involved in merges so they have more time for
> the leak to accumulate.
> *Solution*
> The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
> use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
> altogether. Given that slowFileExists doesn't care about the file content at
> all it seems wasteful to even create a mmap for it in the first place and
> there's probably a more efficient way to check its existence.
> Lucene's arena-pooling also gets much better in 10.x, as they tweak some
> defaults related to this leak behavior. We should consider adopting those
> changes in 9.x by setting
> {{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
> Lucene default in 10.x due to all the problems this pooling behavior raised
> in their 9.12.x line). See
> [here|https://github.com/apache/lucene/issues/15068] for more context on that
> sysprop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
