[
https://issues.apache.org/jira/browse/LUCENE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407363#comment-17407363
]
Michael McCandless commented on LUCENE-9969:
--------------------------------------------
Imagine we had a {{NUMERIC}} doc values field, holding the parent ordinal of
each ordinal in the taxonomy index. I think we can easily create that while
indexing, since we already ensure a parent is assigned an ordinal before its
children.
Then, at search time, instead of using the big non-sparse hard-allocated
{{int[] parents}} array, we could pull a {{NumericDocValues}} iterator, sort
the ordinals we had just counted (the bitset idea from LUCENE-10080 might help
with that?), and make a single iteration through the DV iterator to find all
parent ordinals, to then know how to collate the ordinals into each dimension?
Except for the added sort (N * log(N) worst case), performance should be good –
doc values are already designed for this forward only iteration. And then we
wouldn't need {{int[] parents}} for "normal" non-hierarchical facet counting.
For truly hierarchical facet counting I'm not sure what to do yet :)
> DirectoryTaxonomyReader.taxoArray占用内存较大导致系统OOM宕机
> ------------------------------------------------
>
> Key: LUCENE-9969
> URL: https://issues.apache.org/jira/browse/LUCENE-9969
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/facet
> Affects Versions: 6.6.2
> Reporter: FengFeng Cheng
> Priority: Trivial
> Attachments: image-2021-05-24-13-43-43-289.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> 首先数据量很大，jvm内存为90G，但是TaxonomyIndexArrays几乎占走了一半
> !image-2021-05-24-13-43-43-289.png!
> 请问对于TaxonomyReader是否有更好的使用方式或者其他的优化？
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
