[ https://issues.apache.org/jira/browse/LUCENE-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695400#action_12695400 ]
Michael McCandless commented on LUCENE-1582: -------------------------------------------- bq. I can add a setValue() method to the tokenStream that resets it with the new value. That's a good step forward, but it'd likely mean the default is to be slower performance? In general I prefer (when realistic) to have default ootb experience to be good performance, but in this case it doesn't seem like there's an easy way to have a natural high-performance default. And eg we don't reuse Document & Field by default, so expecting someone to do a bit of work to reuse Trie's TokenStreams seems OK. It's almost like.... Analyzer.reusableTokenStream(...) should "know" it's deailing with a Numeric field, and handle the reuse for you, in a future world when Lucene knows that a Field is a NumericField, meant to be indexed using trie. But we can leave all of that for future optimization; for now, providing setValue is great. > Make TrieRange completely independent from Document/Field with TokenStream of > prefix encoded values > --------------------------------------------------------------------------------------------------- > > Key: LUCENE-1582 > URL: https://issues.apache.org/jira/browse/LUCENE-1582 > Project: Lucene - Java > Issue Type: Improvement > Components: contrib/* > Affects Versions: 2.9 > Reporter: Uwe Schindler > Assignee: Uwe Schindler > Fix For: 2.9 > > Attachments: LUCENE-1582.patch > > > TrieRange has currently the following problem: > - To add a field, that uses a trie encoding, you can manually add each term > to the index or use a helper method from TrieUtils. The helper method has the > problem, that it uses a fixed field configuration > - TrieUtils currently creates per default a helper field containing the lower > precision terms to enable sorting (limitation of one term/document for > sorting) > - trieCodeLong/Int() creates unnecessarily String[] and char[] arrays that is > heavy for GC, if you index lot of numeric values. Also a lot of char[] to > String copying is involved. > This issue should improve this: > - trieCodeLong/Int() returns a TokenStream. During encoding, all char[] > arrays are reused by Token API, additional String[] arrays for the encoded > result are not created, instead the TokenStream enumerates the trie values. > - Trie fields can be added to Documents during indexing using the standard > API: new Field(name,TokenStream,...), so no extra util method needed. By > using token filters, one could also add payload and so and customize > everything. > The drawback is: Sorting would not work anymore. To enable sorting, a > (sub-)issue can extend the FieldCache to stop iterating the terms, as soon as > a lower precision one is enumerated by TermEnum. I will create a "hack" patch > for TrieUtils-use only, that uses a non-checked Exceptionin the Parser to > stop iteration. With LUCENE-831, a more generic API for this type can be used > (custom parser/iterator implementation for FieldCache). I will attach the > field cache patch (with the temporary solution, until FieldCache is > reimplemented) as a separate patch file, or maybe open another issue for it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org