That's expected. Non letters are not mapped to letters, correctly. On Oct 19, 2017 9:38 AM, "Chitra" <chithu.r...@gmail.com> wrote:
> Hi, > I indexed a term 'ⒶeŘꝋꝒɫⱯŋɇ' (aeroplane) and the term was > indexed as "er l n", some characters were trimmed while indexing. > > Here is my code > > protected Analyzer.TokenStreamComponents createComponents(final String > > fieldName, final Reader reader) > > { > > final ClassicTokenizer src = new ClassicTokenizer(getVersion(), > > reader); > > src.setMaxTokenLength(ClassicAnalyzer.DEFAULT_MAX_TOKEN_LENGTH); > > > > TokenStream tok = new ClassicFilter(src); > > tok = new LowerCaseFilter(getVersion(), tok); > > tok = new StopFilter(getVersion(), tok, stopwords); > > tok = new ASCIIFoldingFilter(tok); // to enable AccentInsensitive > > search > > > > return new Analyzer.TokenStreamComponents(src, tok) > > { > > @Override > > protected void setReader(final Reader reader) throws > > IOException > > { > > > > src.setMaxTokenLength(ClassicAnalyzer.DEFAULT_MAX_TOKEN_LENGTH); > > super.setReader(reader); > > } > > }; > > } > > > > Am I missing anything? Is that expected behavior for my input or any reason > behind such abnormal behavior? > > -- > Regards, > Chitra >