[
https://issues.apache.org/jira/browse/SOLR-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski updated SOLR-18551:
-----------------------------------
Description:
MMapDirectory works by mapping index files into a process's virtual address
space, but deleting these maps later on can an expensive operation for the JVM.
Lucene, starting in 9.12, has dealt with this expense by lumping mappings
together in "arenas" which can then be unmapped in bulk later. This changes
the mmap lifecycle a bit: individual mmaps are added to the arena on creation
and now persist until the entire arena is closed at some point later (typically
when the DirectoryReader itself is closed).
When DirectoryReaders and SegmentReaders (and the arena's they own) are
long-lived, these mmaps can accumulate and eventually exhaust the OS mmap count
or even the virtual-memory for a Solr process. This doesn't happen normally
because IndexWriter, IndexReader, etc. are conscientious in the files that they
open, and because SegmentReaders don't tend to be open all that long.
But this has caused leaks in Lucene
([example|https://github.com/apache/lucene/issues/15068]) and I've found a leak
scenario in Solr 9.10.1 now too.
*Solr Leak Scenario*
The particular path I've seen that triggers this leak in Solr is around the
periodic polling done by TLOG and PULL replicas.
Each time a TLOG or PULL replica polls the leader and there are new changes,
the leader will provide a list of its index files and the follower will call
"[IndexFetcher.slowFileExists|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1337-L1344]";
to see which of the files are already present locally. "slowFileExists" calls
{{Directory.openInput(fileName, IOContext.DEFAULT)}} as a quick way to see if
the file is present. But when MMapDirectory is in use, this tiny existence
check creates a (possibly large) mmap that may stick around for quite awhile -
often until the segment is merged away and the SegmentReader/arena are closed.
This existence-check leak happens for each leader index file that exists on the
follower and repeats every polling cycle. I've seen this hit the processes
mmap limit. I've also seen it exhaust the process's virtual memory space
(128TB!) with a comparatively small index of ~100gb.
The issue tends to be worse on large indices that receive a trickle of updates:
both because they have more index files and because they have many larger index
files that are infrequently involved in merges.
*Solution*
The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
altogether. Given that slowFileExists doesn't care about the file content at
all it seems wasteful to even create a mmap for it in the first place and
there's probably a more efficient way to check its existence.
Lucene's arena-pooling also gets much better in 10.x, as they tweak some
defaults related to this leak behavior. We should consider adopting those
changes in 9.x by setting
{{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
Lucene default in 10.x due to all the problems this pooling behavior raised in
their 9.12.x line). See [here|https://github.com/apache/lucene/issues/15068]
for more context on that sysprop.
was:
MMapDirectory works by mapping index files into a process's virtual address
space, but deleting these maps later on can an expensive operation for the JVM.
Lucene, starting in 9.12, has dealt with this expense by lumping mappings
together in "arenas" which can then be unmapped in bulk later. This changes
the mmap lifecycle a bit: individual mmaps are added to the arena on creation
and now persist until the entire arena is closed at some point later (typically
when the DirectoryReader itself is closed).
When DirectoryReaders and SegmentReaders (and the arena's they own) are
long-lived, these mmaps can accumulate and eventually exhaust the OS mmap count
or even the virtual-memory for a Solr process. This doesn't happen normally
because IndexWriter, IndexReader, etc. are conscientious in the files that they
open, and because SegmentReaders don't tend to be open all that long.
But this has caused leaks in Lucene
([example|https://github.com/apache/lucene/issues/15068]) and I've found a leak
scenario in Solr 9.10.1 now too.
*Solr Leak Scenario*
The particular path I've seen that triggers this leak in Solr is around the
periodic polling done by TLOG and PULL replicas.
Each time a TLOG or PULL replica polls the leader and there are new changes,
the leader will provide a list of its index files and the follower will call
"IndexFetcher.slowFileExists" to see which of the files are already present
locally. "slowFileExists" calls {{Directory.openInput(fileName,
IOContext.DEFAULT)}} as a quick way to see if the file is present. But when
MMapDirectory is in use, this tiny existence check creates a (possibly large)
mmap that may stick around for quite awhile - often until the segment is merged
away and the SegmentReader/arena are closed.
This existence-check leak happens for each leader index file that exists on the
follower and repeats every polling cycle. I've seen this hit the processes
mmap limit. I've also seen it exhaust the process's virtual memory space
(128TB!) with a comparatively small index of ~100gb.
The issue tends to be worse on large indices that receive a trickle of updates:
both because they have more index files and because they have many larger index
files that are infrequently involved in merges.
*Solution*
The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
altogether. Given that slowFileExists doesn't care about the file content at
all it seems wasteful to even create a mmap for it in the first place and
there's probably a more efficient way to check its existence.
Lucene's arena-pooling also gets much better in 10.x, as they tweak some
defaults related to this leak behavior. We should consider adopting those
changes in 9.x by setting
{{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
Lucene default in 10.x due to all the problems this pooling behavior raised in
their 9.12.x line). See [here|https://github.com/apache/lucene/issues/15068]
for more context on that sysprop.
> IndexFetcher can leak "mmaps" on TLOG/PULL non-leaders
> ------------------------------------------------------
>
> Key: SOLR-18551
> URL: https://issues.apache.org/jira/browse/SOLR-18551
> Project: Solr
> Issue Type: Bug
> Affects Versions: 10.0, main(11.0), 9.10.1
> Reporter: Jason Gerlowski
> Priority: Major
>
> MMapDirectory works by mapping index files into a process's virtual address
> space, but deleting these maps later on can an expensive operation for the
> JVM. Lucene, starting in 9.12, has dealt with this expense by lumping
> mappings together in "arenas" which can then be unmapped in bulk later. This
> changes the mmap lifecycle a bit: individual mmaps are added to the arena on
> creation and now persist until the entire arena is closed at some point later
> (typically when the DirectoryReader itself is closed).
> When DirectoryReaders and SegmentReaders (and the arena's they own) are
> long-lived, these mmaps can accumulate and eventually exhaust the OS mmap
> count or even the virtual-memory for a Solr process. This doesn't happen
> normally because IndexWriter, IndexReader, etc. are conscientious in the
> files that they open, and because SegmentReaders don't tend to be open all
> that long.
> But this has caused leaks in Lucene
> ([example|https://github.com/apache/lucene/issues/15068]) and I've found a
> leak scenario in Solr 9.10.1 now too.
> *Solr Leak Scenario*
> The particular path I've seen that triggers this leak in Solr is around the
> periodic polling done by TLOG and PULL replicas.
> Each time a TLOG or PULL replica polls the leader and there are new changes,
> the leader will provide a list of its index files and the follower will call
> "[IndexFetcher.slowFileExists|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1337-L1344]";
> to see which of the files are already present locally. "slowFileExists"
> calls {{Directory.openInput(fileName, IOContext.DEFAULT)}} as a quick way to
> see if the file is present. But when MMapDirectory is in use, this tiny
> existence check creates a (possibly large) mmap that may stick around for
> quite awhile - often until the segment is merged away and the
> SegmentReader/arena are closed.
> This existence-check leak happens for each leader index file that exists on
> the follower and repeats every polling cycle. I've seen this hit the
> processes mmap limit. I've also seen it exhaust the process's virtual memory
> space (128TB!) with a comparatively small index of ~100gb.
> The issue tends to be worse on large indices that receive a trickle of
> updates: both because they have more index files and because they have many
> larger index files that are infrequently involved in merges.
> *Solution*
> The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
> use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
> altogether. Given that slowFileExists doesn't care about the file content at
> all it seems wasteful to even create a mmap for it in the first place and
> there's probably a more efficient way to check its existence.
> Lucene's arena-pooling also gets much better in 10.x, as they tweak some
> defaults related to this leak behavior. We should consider adopting those
> changes in 9.x by setting
> {{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
> Lucene default in 10.x due to all the problems this pooling behavior raised
> in their 9.12.x line). See
> [here|https://github.com/apache/lucene/issues/15068] for more context on that
> sysprop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
