[
https://issues.apache.org/jira/browse/SOLR-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
abhishek bafna updated SOLR-7193:
---------------------------------
Comment: was deleted
(was: The ConcatenateWordsFilter takes all the input token (words) and generate
a single token. The CPU time and memory depends on the number and size of the
tokens coming in the stream. The use case for this filter, when input stream
contains business name, address, etc., which usually have a small number of
tokens. I am guessing, here (test environment) input data containing long
paragraphs or documents and that might be causing the issue.)
> Concatenate words from token stream
> -----------------------------------
>
> Key: SOLR-7193
> URL: https://issues.apache.org/jira/browse/SOLR-7193
> Project: Solr
> Issue Type: New Feature
> Components: Schema and Analysis
> Reporter: abhishek bafna
> Attachments: concatenate_words.patch
>
>
> The user entered data often don't have proper spacing between words and words
> spelling and format also varies from data like business names, address etc.
> After tokenizing data, we might perform pattern replacement, stop word
> filtering etc. Later we want to concatenate all the tokens and generate
> n-grams token for indexing business name and perform the fuzzy match.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
