[
https://issues.apache.org/jira/browse/OAK-12193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Mueller reopened OAK-12193:
----------------------------------
> Large numbers of deleted Lucene documents cause incremental indexing to
> consume too much memory
> -----------------------------------------------------------------------------------------------
>
> Key: OAK-12193
> URL: https://issues.apache.org/jira/browse/OAK-12193
> Project: Jackrabbit Oak
> Issue Type: Improvement
> Components: indexing, lucene
> Reporter: Thomas Mueller
> Assignee: Thomas Mueller
> Priority: Major
> Fix For: 2.2.0
>
>
> When Oak performs an incremental indexing cycle which contains a lot of
> deletions, it can consume a lot of memory.
> The issue is caused by accumulated Lucene document deletions, which are sent
> to each of the Lucene Index writers for all the indexes that cover the path
> of the deleted node (regardless of whether they match the nodetype of it).
> Since there can be numerous indexes the cumulative memory use can easily
> overwhelm the JVM.
> {noformat}
> Example thread:
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.MultiplexingIndexWriter
>
> org.apache.jackrabbit.oak.plugins.index.search.CompositePropertyUpdateCallback
>
> org.apache.jackrabbit.oak.plugins.index.search.spi.binary.FulltextBinaryTextExtractor
>
> org.apache.jackrabbit.oak.plugins.index.lucene.directory.DefaultDirectoryFactory
>
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriterFactory
>
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenelndexEditorContext
> Thread name: async-index-update-async
> memory bytes percentage class @ instance
> 2,516,736,256 69.87% java.lang.Thread @ 0x6a52565d0 async-index-update-async
> Thread
> 879,048,256 24.41%
> org.apache.jackrabbit.oak.plugins.index.lucene.LucenelndexEditorContext @
> 0x6be46e860
> 879,047,904 24.41%
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.MultiplexingIndexWriter
> @ 0x6be46e8e0
> 879,047,864 24.41% java.util.HashMap @ 0x6be46e908
> 879,047,816 24.41% • java.util.HashMap$Node[16] @ 0x6be46e938
> 879,047,736 24.41% java.util.HashMap$Node @ 0x6be46e988
> 879,047,704 24.41%
> org.apache.jackrabbit.oak.plugins.index.lucene.writer.DefaultIndexWriter @
> 0x6be46e9a8
> 879,047,648 24.41% org.apache.lucene.index.IndexWriter @ 0x6be46e9e0
> 870,569,576 24.17%
> org.apache.lucene.index.ThreadAffinityDocumentsWriterThreadPool @ 0x6be477680
> 870,568,880 24.17%
> org.apache.lucene.index.DocumentsWriterPerThreadPool$ThreadState @ 0x6be477888
> 870,568,816 24.17% org.apache.lucene.index.DocumentsWriterPerThread @
> 0x6be4778c8
> 863,858,200 23.98%
> org.apache.lucene.index.DocumentsWriterDeleteQueue$DeleteSlice @ 0x6be4f4020
> 4,381,272 0.12% org.apache.lucene.index.BufferedUpdates @ 0x6be4b3978
> 2,325,704 0.06% org.apache.lucene.index.DocFieldProcessor @ 0x6be477a50
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
