[jira] [Commented] (LUCENE-10396) Automatically create sparse indexes for sort fields

[Ignacio Vera (Jira)]

Title: Message Title

Ignacio Vera commented on LUCENE-10396   Re: Automatically create sparse indexes for sort fields Here is the code I am using when using postings/TermsEnum from the inverted index which might be totally wrong / inefficient as unfortunately I am not an expert in this area: // position the terms enum on the current term termsEnum.seekExact(sortedDocValues.lookupOrd(sortedDocValues.ordValue())); // advance if (termsEnum.next() == null) { doc = sortedDocValues.advance(DocIdSetIterator.NO_MORE_DOCS); } else { termsEnum.postings(postingsEnum); doc = sortedDocValues.advance(postingsEnum.nextDoc()); } This code performs ok for lower cardinality but it becomes slow for high cardinality. Similar to what I have done in the linked PR, I have indexed 50 million documents in a sorted index. The documents contain a SortedDocValues with a 10 bytes term and the term is indexed using a StringField as well. I checked the index size and the speed of retrieving the first document per term with different cardinalities and the results looks like: 1000 cardinality INDEX SIZE: 6.110762596130371 MB Average: 0.0026419270000000005 seconds 10000 cardinality INDEX SIZE: 22.32368278503418 MB Average: 0.018914526950000003 seconds 100000 cardinality INDEX SIZE: 86.16338920593262 MB Average: 0.1449143188 seconds 500000 cardinality INDEX SIZE: 108.61055660247803 MB Average: 0.44313388750000005 seconds Add Comment   This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)