I have a CustomAnalyzer which overrides "public final TokenStream tokenStream (
String fieldName, Reader reader )":
@Override
public final TokenStream tokenStream ( String fieldName, Reader reader )
{
boolean fieldRequiresExactMatching =
IndexManager.getInstance().isExactMatchField( fieldName );
Reader localreader = reader;
if ( !fieldRequiresExactMatching )
{
NormalizeCharMap charMap = new NormalizeCharMap();
charMap.add(",", " ");
<SNIP>
// wrap/filter reader
localreader = new MappingCharFilter( charMap, reader );
}
TokenStream t = new WhitespaceAnalyzer( IndexManager.CURRENT_LUCENE_VERSION
).tokenStream( fieldName, localreader );
if ( !fieldRequiresExactMatching )
{
// apply stop word filter
Set<String> stopWordSet = null;
<SNIP>
if ( stopWordSet != null )
{
// wrap/filter stream
StopFilter stopFilter = new StopFilter(
IndexManager.CURRENT_LUCENE_VERSION, t, stopWordSet, true );
t = stopFilter;
}
}
return t;
}
MappingCharFilter -> whiteSpace analysis - <if condition given> -> stop word
filtering
As of Lucene 4.0 " protected TokenStreamComponents createComponents ( final
String fieldName, final Reader reader )" is to be overridden and a
TokenStreamComponents has tob e returned. I don't see how to achieve this ...
all I have is a TokenStream but no Tokenizer ...