On Mar 8, 2011, at 9:36 AM, Marvin Humphrey wrote: > Greets, > > Right now, Lucy only has one tokenizer-style Analyzer subclass: > Lucy::Analysis::Tokenizer, which is regex based. > > At some point, I expect we will have other tokenizer classes which don't use a > regex engine, so I think it would be best to reserve the name "Tokenizer" for > future use and rename the current Tokenizer to "RegexTokenizer".
Lucy::Tokenizer::Regex please! Best, David
