[ https://issues.apache.org/jira/browse/LUCENE-10023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377413#comment-17377413 ]
Michael Gibney commented on LUCENE-10023: ----------------------------------------- In contrast to "naive word cloud" faceting, the more compelling use cases for multi-token post-analysis DocValues tend to be specialized cases, with inherent limitations on the number of tokens. A couple of comments on related issues mention {{path_tokenizer}} in Elasticsearch (see [this comment|https://github.com/elastic/elasticsearch/issues/12394#issuecomment-199555310], and [this comment|https://github.com/elastic/elasticsearch/issues/18064#issuecomment-232297988]). My own use case has to do with fields that are in a sense single-valued, but with TokenFilters that may produce expanded "synonym"-style mappings (really broader/narrower/related/hierarchical entities). And fwiw, I would argue that there are legitimate use cases even for the "naive word cloud" approach -- text corpus analytics, etc. I realize that it would be possible to do this work external to Lucene; but to me it felt cleanest to add it here, at least to have something concrete for seeding discussion. The PR initially includes only a trivial test demonstrating the new behavior; more tests can be added if there's a decision to further pursue this approach. > Multi-token post-analysis DocValues > ----------------------------------- > > Key: LUCENE-10023 > URL: https://issues.apache.org/jira/browse/LUCENE-10023 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index > Reporter: Michael Gibney > Priority: Major > > The single-token case for post-analysis DocValues is accounted for by > {{Analyzer.normalize(...)}} (and formerly {{MultiTermAwareComponent}}); but > there are cases where it would be desirable to have post-analysis DocValues > based on multi-token fields. > The main use cases that I can think of are variants of faceting/terms > aggregation. I understand that this could be viewed as "trappy" for the naive > "Moby Dick word cloud" case; but: > # I think this can be supported fairly cleanly in Lucene > # Explicit user configuration of this option would help prevent people > shooting themselves in the foot > # The current situation is arguably "trappy" as well; it just offloads the > trappiness onto Lucene-external workarounds for systems/users that want to > support this kind of behavior > # Integrating this functionality directly in Lucene would afford consistency > guarantees that present opportunities for future optimizations (e.g., shared > Terms dictionary between indexed terms and DocValues). > This issue proposes adding support for multi-token post-analysis DocValues > directly to {{IndexingChain}}. The initial proposal involves extending the > API to include {{IndexableFieldType.tokenDocValuesType()}} (in addition to > existing {{IndexableFieldType.docValuesType()}}). -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org