I have a CustomAnalyzer which overrides "public final TokenStream tokenStream ( 
String fieldName, Reader reader )":
@Override
public final TokenStream tokenStream ( String fieldName, Reader reader )
{
boolean fieldRequiresExactMatching = 
IndexManager.getInstance().isExactMatchField( fieldName );

Reader localreader = reader;
if ( !fieldRequiresExactMatching )
{
        NormalizeCharMap charMap = new NormalizeCharMap();
        charMap.add(",", " ");
<SNIP>
        // wrap/filter reader
        localreader = new MappingCharFilter( charMap, reader );                 
}
TokenStream t = new WhitespaceAnalyzer( IndexManager.CURRENT_LUCENE_VERSION 
).tokenStream( fieldName, localreader );

if ( !fieldRequiresExactMatching )
{
        // apply stop word filter
        Set<String> stopWordSet = null;
<SNIP>
        if ( stopWordSet != null )
        {
                // wrap/filter stream
                StopFilter stopFilter = new StopFilter( 
IndexManager.CURRENT_LUCENE_VERSION, t, stopWordSet, true );
                t = stopFilter;
        }
}
return t;
}

MappingCharFilter -> whiteSpace analysis - <if condition given> -> stop word 
filtering

As of Lucene 4.0 " protected TokenStreamComponents createComponents ( final 
String fieldName, final Reader reader )" is to be overridden and  a 
TokenStreamComponents has tob e returned. I don't see how to achieve this ... 
all I have is a TokenStream but no Tokenizer ...

Reply via email to