[
https://issues.apache.org/jira/browse/LUCENE-10023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377594#comment-17377594
]
Michael Gibney commented on LUCENE-10023:
-----------------------------------------
{quote}pushing the trappiness onto them is a good thing
{quote}
I think that's a reasonable perspective. That said, I was motivated to add the
functionality here because I see the same questions being raised in
Elasticsearch and Solr forums (and presumably it'd be useful in other contexts
as well); and it's a discrete enough change that I was looking to factor out
the common functionality to the appropriate place in the software stack.
{quote}I'm especially worried about search servers that would expose this
blindly without such checks, and using configured analyzers that wouldn't have
limit-filters and so on.
{quote}
I could see this concern leading one way or the other: such a default limit
(perhaps configurable via FieldType?) could be enforced with negligible
overhead directly in IndexingChain, protecting users against even a buggy
TokenStream. I definitely see how the careless use of this feature could be
problematic; but even _without_ built-in limits, assuming this feature is not
enabled by default, a user would have to explicitly enable it -- which they
would presumably only do in response to a specific need for the functionality
this feature supports.
{quote}The current patch really just shoves sugar into indexwriter [....] It
isn't any more efficient than the user consuming the TS themselves and adding
the field
{quote}
Wouldn't it be more efficient though, in at least some cases? With an
"external" approach such as you're suggesting, a tokenized and indexed field
would have to run analysis twice: once outside of IndexingChain in order to
generate (and buffer) explicit SortedSetDocValuesFields, and again within
IndexingChain to generate indexed tokens (with associated token attributes). In
contrast, the current PR only runs analysis once, and avoids buffering
BytesRefs (i.e., via deepCopy() -- also avoids creating disposable one-off
SortedSetDocValuesField objects), and instead sends BytesRefs directly to the
docValuesWriter.
Aside from performance, the approach taken by this PR is more than sugar in the
sense that an IndexingChain-internal approach can enforce consistency between
indexed terms and docValues terms, which could be useful in a number of ways
(e.g., reliable behavior in applications that assume 1:1 correspondence between
indexed terms and docValues, and potential future optimizations such as a
shared terms dictionary).
> Multi-token post-analysis DocValues
> -----------------------------------
>
> Key: LUCENE-10023
> URL: https://issues.apache.org/jira/browse/LUCENE-10023
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael Gibney
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The single-token case for post-analysis DocValues is accounted for by
> {{Analyzer.normalize(...)}} (and formerly {{MultiTermAwareComponent}}); but
> there are cases where it would be desirable to have post-analysis DocValues
> based on multi-token fields.
> The main use cases that I can think of are variants of faceting/terms
> aggregation. I understand that this could be viewed as "trappy" for the naive
> "Moby Dick word cloud" case; but:
> # I think this can be supported fairly cleanly in Lucene
> # Explicit user configuration of this option would help prevent people
> shooting themselves in the foot
> # The current situation is arguably "trappy" as well; it just offloads the
> trappiness onto Lucene-external workarounds for systems/users that want to
> support this kind of behavior
> # Integrating this functionality directly in Lucene would afford consistency
> guarantees that present opportunities for future optimizations (e.g., shared
> Terms dictionary between indexed terms and DocValues).
> This issue proposes adding support for multi-token post-analysis DocValues
> directly to {{IndexingChain}}. The initial proposal involves extending the
> API to include {{IndexableFieldType.tokenDocValuesType()}} (in addition to
> existing {{IndexableFieldType.docValuesType()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
