[
https://issues.apache.org/jira/browse/LUCENE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350589#comment-17350589
]
Gautam Worah commented on LUCENE-9969:
--------------------------------------
If I understand things correctly from Google Translate you are trying to dig
into why the collective {{TaxonomyIndexArrays}} (parent, child and sibling
arrays) are taking about 60% of your JVM heap.
I suspect that the cardinality in your data is very high i.e, the number of
unique taxonomy ordinals is very high causing the parent, children and sibling
arrays to be massive which in turn occupy space in your heap.
One recommendation that I have is to try to bucket values and reduce this
cardinality. Does your application need counts for all the individual values or
can it work with buckets or "coarser" values. This might significantly reduce
the number of labels in your index and consequently the number of ordinals.
You can even try other techniques of finding very optimized buckets (not just
uniform ones). Anything that reduces cardinality will reap huge benefits in
terms of ordinal array size.
Otherwise, maybe you could try upgrading Lucene to its latest release (8.8)?
Newer versions of Lucene taxonomy faceting have
[switched|https://github.com/apache/lucene-solr/pull/1733] to
{{BinaryDocValues}} instead of {{StoredFields}} and generally have a lot of
useful improvements that help with performance.
It might help to provide more information into how your application uses
faceting or the type of data it uses...
> DirectoryTaxonomyReader.taxoArray占用内存较大导致系统OOM宕机
> ------------------------------------------------
>
> Key: LUCENE-9969
> URL: https://issues.apache.org/jira/browse/LUCENE-9969
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/facet
> Affects Versions: 6.6.2
> Reporter: FengFeng Cheng
> Priority: Trivial
> Attachments: image-2021-05-24-13-43-43-289.png
>
>
> 首先数据量很大，jvm内存为90G，但是TaxonomyIndexArrays几乎占走了一半
> !image-2021-05-24-13-43-43-289.png!
> 请问对于TaxonomyReader是否有更好的使用方式或者其他的优化？
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
