[ https://issues.apache.org/jira/browse/LUCENE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143219#comment-14143219 ]
Michael McCandless commented on LUCENE-5879: -------------------------------------------- bq. So for the whole segment, does it decide to insert auto-prefix'es at specific byte lengths (e.g. 3, 5, and 7)? Or does it vary based on specific terms? I'm hoping it's smart enough to vary based on specific terms. For example if, hypothetically there were lots of terms that had this common prefix: "BGA" then it might decide "BGA" makes a good auto-prefix but not necessarily all terms at length 3 since many others might not make good prefixes. Make sense? It's dynamic, based on how terms occupy the space. Today (and we can change this: it's an impl. detail) it assigns prefixes just like it assigns terms to blocks. Ie, when it sees a given prefix matches 25 - 48 terms, it inserts a new auto-prefix term. That auto-prefix term replaces those 25 - 48 other terms with 1 term, and then the process "recurses", i.e. you can then have a shorter auto-prefix term matching 25 - 48 other normal and/or auto-prefix terms. bq. At a low level, do I take advantage of this in the same way that I might do so at a high level using PrefixQuery and then getting the weight then getting the scorer to iterate docIds? Or is there a lower-level path? Although there is some elegance to not introducing new APIs, I think it's worth exploring having prefix & range capabilities be on the TermsEnum in some way. What this patch does is generalize/relax Terms.intersect: that method no longer guarantees that you see "true" terms. Rather, it now guarantees only that the docIDs you visit will be the same. So to take advantage of it, you need to pass an Automaton to Terms.intersect and then not care about which terms you see, only the docIDs after visiting the DocsEnum across all terms it returned to you. bq. Do you envision other posting formats being able to re-use the logic here? That would be nice. I agree it would be nice ... and the index-time logic that identifies the auto-prefix terms is quite standalone, so e.g. we could pull it out and have it "wrap" the incoming Fields to insert the auto-prefix terms. This way it's nearly transparent to any postings format ... But the problem is, at search time, there's tricky logic in intersect() to use these prefix terms ... factoring that out so other formats can use it is trickier I think... though maybe we could fold it into the default Terms.intersect() impl... bq. In your future tuning, I suggest you give the ability to vary the conservative vs aggressive prefixing based on the very beginning and very end (assuming known common lengths). In the FlexPrefixTree Varun (GSOC) worked on, the leaves per level is configurable at each level (i.e. prefix length)... and it's better to have little prefixing at the very top and little at the bottom too. At the top, prefixes only help for queries span massive portions of the possible term space (which in spatial is rare; likely other apps too). And at the bottom (long prefixes) just shy of the maximum length (say 7 bytes out of 8 for a double), there is marginal value because in the spatial search algorithm, the bottom detail is scan'ed over (e.g. TermsEnum.next()) instead of seek'ed, because the data is less dense and it's adjacent. This principle may apply to numeric-range queries depending on how they are coded; I'm not sure. I agree this (how auto-prefix terms are assigned) needs more control / experimenting. Really the best prefixes are a function not only of how the terms were distributed, but also of how queries will "likely" ask for ranges. I think it's similar to postings skip lists, where we have different frequency of a skip pointer on the "leaf" level vs the "upper" skip levels. > Add auto-prefix terms to block tree terms dict > ---------------------------------------------- > > Key: LUCENE-5879 > URL: https://issues.apache.org/jira/browse/LUCENE-5879 > Project: Lucene - Core > Issue Type: New Feature > Components: core/codecs > Reporter: Michael McCandless > Assignee: Michael McCandless > Fix For: 5.0, Trunk > > Attachments: LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, > LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch > > > This cool idea to generalize numeric/trie fields came from Adrien: > Today, when we index a numeric field (LongField, etc.) we pre-compute > (via NumericTokenStream) outside of indexer/codec which prefix terms > should be indexed. > But this can be inefficient: you set a static precisionStep, and > always add those prefix terms regardless of how the terms in the field > are actually distributed. Yet typically in real world applications > the terms have a non-random distribution. > So, it should be better if instead the terms dict decides where it > makes sense to insert prefix terms, based on how dense the terms are > in each region of term space. > This way we can speed up query time for both term (e.g. infix > suggester) and numeric ranges, and it should let us use less index > space and get faster range queries. > > This would also mean that min/maxTerm for a numeric field would now be > correct, vs today where the externally computed prefix terms are > placed after the full precision terms, causing hairy code like > NumericUtils.getMaxInt/Long. So optos like LUCENE-5860 become > feasible. > The terms dict can also do tricks not possible if you must live on top > of its APIs, e.g. to handle the adversary/over-constrained case when a > given prefix has too many terms following it but finer prefixes > have too few (what block tree calls "floor term blocks"). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org