[ https://issues.apache.org/jira/browse/LUCENE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hiroaki Kawai updated LUCENE-1218: ---------------------------------- Attachment: PassTokenizer.java > PassTokenizerFilter that pass text in a Token > --------------------------------------------- > > Key: LUCENE-1218 > URL: https://issues.apache.org/jira/browse/LUCENE-1218 > Project: Lucene - Java > Issue Type: New Feature > Components: Analysis > Reporter: Hiroaki Kawai > Priority: Minor > Attachments: PassTokenizer.java > > > The PassTokenizer passes a text in a TokenStream that has a single token in > its stream. > This Tokenizer(attached) is very useful for debugging. You may test > TokenFilter that will do sub-tokenization in a token. This Tokenizer is also > useful for debugging the tokenization process dependent on the TokenFilter > order. > This Tokenizer is not suitable for processing a large text field. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]