Re: LetterTokenizer to allow digits

Peter Pimley wrote: Hi everybody, I have just found myself in the situation of having to subclass CharTokenizer with a class that tests against Character.isLetterOrDigit. I would use a LetterTokenizer, but it's important for me to allow numbers through, as the documents I'm indexing often have

LetterTokenizer to allow digits

Hi everybody, I have just found myself in the situation of having to subclass CharTokenizer with a class that tests against Character.isLetterOrDigit. I would use a LetterTokenizer, but it's important for me to allow numbers through, as the documents I'm indexing often have dates such as '2000