The Char-filters MUST come before the Tokenizer, due to their nature of 
processing the character-stream and not the tokens.

If you need to apply the accent normalizatino later in the analysis chain, 
either use ISOLatin1AccentFilterFactory or help with the implementation of 
SOLR-1978.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
Training in Europe - www.solrtraining.com

On 5. juli 2010, at 17.32, Saïd Radhouani wrote:

> Thanks Koji for the reply and for updating wiki. As it's written now in wiki, 
> it sounds (at least to me) like MappingCharFilterFactory works only with 
> WhitespaceTokenizerFactory.
> 
> Did you really mean that? Because this filter  works also with other 
> tkenizers. For instance, in my text type, I'm using StandardTokenizerFactory 
> for document processing, and  WhitespaceTokenizerFactory for query processing.
> 
> I also noticed that, in whatever order you put this filter in the definition 
> of a field type, it's always applied (during text processing) before the 
> tokenizer and all the other filters. Is there a reason for that? Is there a 
> possibility to force the filter to be applied at a certain order among the 
> other filters?
> 
> Thanks,
> -S
> 
> On Jul 5, 2010, at 4:28 PM, Koji Sekiguchi wrote:
> 
>> 
>>> In the same wiki, they say that CharStreamAwareWhitespaceTokenizerFactory 
>>> must be used with MappingCharFilterFactory. But, when I use these tokenizer 
>>> and filter together, I get a sever error saying that the filed type 
>>> containing these filter and tokenizer is unknown. However, when I use this 
>>> filter with StandardTokenizerFactory  or WhitespaceTokenizerFactory!
>>> 
>>> 
>> The wiki is not correct today. Before Lucene 2.9 (and Solr 1.4),
>> Tokenizers can take Reader argument in constructor. But after that,
>> because they can take CharStream argument in constructor,
>> *CharStreamAware* Tokenizers are no longer needed (all Tokenizers
>> are aware of CharStream). I'll update the wiki.
>> 
>> Koji
>> 
>> -- 
>> http://www.rondhuit.com/en/
>> 
> 

Reply via email to