[ https://issues.apache.org/jira/browse/LUCENE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683723#comment-13683723 ]
Han Jiang commented on LUCENE-5029: ----------------------------------- {quote} Also, at write time, it'd be nice if the PBF got a DataOutput + LongsRef from the terms dict which it could write the term data to, and a "matching" DataInput + LongsRef at read time. I'm not sure these even need to reside in BlockTermState? {quote} Yes, we can move those byte[] and DataI/O into TermState, so that the API will be simpler. But with this change, the concept of 'TermState' needs more explanation & javadocs now. > factor out a generic 'TermState' for better sharing in FST-based term dict > -------------------------------------------------------------------------- > > Key: LUCENE-5029 > URL: https://issues.apache.org/jira/browse/LUCENE-5029 > Project: Lucene - Core > Issue Type: Sub-task > Reporter: Han Jiang > Assignee: Han Jiang > Priority: Minor > Fix For: 4.4 > > Attachments: LUCENE-5029.algebra.patch, LUCENE-5029.patch, > LUCENE-5029.patch, LUCENE-5029.patch > > > Currently, those two FST-based term dict (memory codec & blocktree) all use > FST<BytesRef> as a base data structure, this might not share much data in > parent arcs, since the encoded BytesRef doesn't guarantee that > 'Outputs.common()' always creates a long prefix. > While for current postings format, it is guaranteed that each FP (pointing to > .doc, .pos, etc.) will increase monotonically with 'larger' terms. That > means, between two Outputs, the Outputs from smaller term can be safely > pushed towards root. However we always have some tricky TermState to deal > with (like the singletonDocID for pulsing trick), so as Mike suggested, we > can simply cut the whole TermState into two parts: one part for comparation > and intersection, another for restoring generic data. Then the data structure > will be clear: this generic 'TermState' will consist of a fixed-length > LongsRef and variable-length BytesRef. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org