Hi,
              I indexed a term 'ⒶeŘꝋꝒɫⱯŋɇ' (aeroplane) and the term was
indexed as "er l n", some characters were trimmed while indexing.

Here is my code

protected Analyzer.TokenStreamComponents createComponents(final String
> fieldName, final Reader reader)
>     {
>         final ClassicTokenizer src = new ClassicTokenizer(getVersion(),
> reader);
>         src.setMaxTokenLength(ClassicAnalyzer.DEFAULT_MAX_TOKEN_LENGTH);
>
>         TokenStream tok = new ClassicFilter(src);
>         tok = new LowerCaseFilter(getVersion(), tok);
>         tok = new StopFilter(getVersion(), tok, stopwords);
>         tok = new ASCIIFoldingFilter(tok); // to enable AccentInsensitive
> search
>
>         return new Analyzer.TokenStreamComponents(src, tok)
>         {
>             @Override
>             protected void setReader(final Reader reader) throws
> IOException
>             {
>
> src.setMaxTokenLength(ClassicAnalyzer.DEFAULT_MAX_TOKEN_LENGTH);
>                 super.setReader(reader);
>             }
>         };
>     }



Am I missing anything? Is that expected behavior for my input or any reason
behind such abnormal behavior?

-- 
Regards,
Chitra

Reply via email to