[
https://issues.apache.org/jira/browse/LUCENE-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155699#comment-14155699
]
Robert Muir commented on LUCENE-5879:
-------------------------------------
{quote}
One approach we might be able to take here is to more strongly
separate out the auto-prefix terms from the ordinary terms (e.g., so
they feel like two different fields, in the impl), such that intersect
becomes something like a merge between the postings format's "normal"
intersect, with these auto-prefix terms pulling from a shadow field.
If we did this then other postings formats could also more easily add
auto-prefix handling, and maybe even the default impl for
Terms.intersect could use auto-prefix terms ...
But we can try this later; this is a big enough change already.
{quote}
I agree it can be deferred, but I am worried if it gets dropped completely.
Here, in order to make this easy to use, the feature is "temporarily" placed in
fieldinfos with a warning.
However, this is kind of screwed up when its currently a blocktree impl detail,
and, well,
its kinda complicated stuff to implement (no offense!). Realistically, if in 3
months, blocktree is still
the only impl, how will people feel if I want to nuke this fieldinfo and
fieldtype?
Will they be -1 that i break spatial/solr/whatever starts using it (possibly
even their source code compile)?
Because I think the following situations are ok:
Result #1
* fieldinfo option (so its a "lucene feature" not a codec-specific feature)
* "generic" impl so that all codecs actually support said feature, like
everything else in fieldinfo
Result #2
* codec-specific option (so its just a blocktree option)
But the following is not an ok "permanent situation":
Result #3
* fieldinfo option (so it advertises as a "lucene feature", not a
codec-specific feature)
* however, only blocktree actually supports the option
* impossible to move on to next-gen termsdict because it doesnt support it
* impossible to remove from fieldinfo because e.g. .document
api/spatial/solr/old indexes/whatever rely upon it
How can we prevent this from happening?
> Add auto-prefix terms to block tree terms dict
> ----------------------------------------------
>
> Key: LUCENE-5879
> URL: https://issues.apache.org/jira/browse/LUCENE-5879
> Project: Lucene - Core
> Issue Type: New Feature
> Components: core/codecs
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 5.0, Trunk
>
> Attachments: LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch,
> LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch, LUCENE-5879.patch,
> LUCENE-5879.patch, LUCENE-5879.patch
>
>
> This cool idea to generalize numeric/trie fields came from Adrien:
> Today, when we index a numeric field (LongField, etc.) we pre-compute
> (via NumericTokenStream) outside of indexer/codec which prefix terms
> should be indexed.
> But this can be inefficient: you set a static precisionStep, and
> always add those prefix terms regardless of how the terms in the field
> are actually distributed. Yet typically in real world applications
> the terms have a non-random distribution.
> So, it should be better if instead the terms dict decides where it
> makes sense to insert prefix terms, based on how dense the terms are
> in each region of term space.
> This way we can speed up query time for both term (e.g. infix
> suggester) and numeric ranges, and it should let us use less index
> space and get faster range queries.
>
> This would also mean that min/maxTerm for a numeric field would now be
> correct, vs today where the externally computed prefix terms are
> placed after the full precision terms, causing hairy code like
> NumericUtils.getMaxInt/Long. So optos like LUCENE-5860 become
> feasible.
> The terms dict can also do tricks not possible if you must live on top
> of its APIs, e.g. to handle the adversary/over-constrained case when a
> given prefix has too many terms following it but finer prefixes
> have too few (what block tree calls "floor term blocks").
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
