[
https://issues.apache.org/jira/browse/SOLR-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157045#comment-13157045
]
Stanislaw Osinski commented on SOLR-2917:
-----------------------------------------
Would a typical TVTokenStream contain stop words, original (unstemmed) forms
and sentence separators? If not, the human-readability of cluster labels would
suffer quite a bit. On the other hand, the schema could define a parallel field
with certain filters disabled, clustering should work nicely with such a
stream. Is there any other solution to this?
> Support for field-specific tokenizers, token- and character filters in search
> results clustering
> ------------------------------------------------------------------------------------------------
>
> Key: SOLR-2917
> URL: https://issues.apache.org/jira/browse/SOLR-2917
> Project: Solr
> Issue Type: Improvement
> Components: contrib - Clustering
> Reporter: Stanislaw Osinski
> Assignee: Stanislaw Osinski
> Fix For: 3.6
>
>
> Currently, Carrot2 search results clustering component creates clusters based
> on the raw text of a field. The reason for this is that Carrot2 aims to
> create meaningful cluster labels by using sequences of words taken directly
> from the documents' text (including stop words: _Development of Lucene and
> Solr_ is more readable than _Development Lucene Solr_). The easiest way of
> providing input for such a process was feeding Carrot2 with raw (stored)
> document content.
> It is, however, possible to take into account +some+ of the field's filters
> during clustering. Because Carrot2 does not currently expose an API for
> feeding pre-tokenized input, the clustering component would need to:
> 1. get raw text of the field,
> 2. run it through the field's char filters, tokenizers and selected token
> filters (omitting e.g. stop words filter and stemmers, Carrot2 needs the
> original words to produce readable cluster labels),
> 3. glue the output back into a string and feed to Carrot2 for clustering.
> In the future, to eliminate step 3, we could modify Carrot2 to accept
> pre-tokenized content.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
